jbarker7 / sqlcc

SQL Code Coverage - SQLCC is a code coverage tool that allows developers the ability to track which lines of code within stored procedures, functions, and triggers have been tested in SQL server by integration tests written by the developer.

sqlcc

Watch this for a quick summary: http://screencast.com/t/5I2N4GQgcb4 (please note: this video's syntax is out of date. Please see below for updated syntax).

The Skinny

SQLCC is a code coverage tool that allows developers the ability to track which lines of code within stored procedures, functions, and triggers have been tested in SQL server by integration tests written by the developer.

The Longer Skinny

We all know it's just silly to stuff business logic in our database-- its harder to maintain, database servers don't scale well, etc. We could go on and on with reasons why you shouldn't put business logic in SQL. However, there are times when you inherit that really ancient code-base. You know, the one with 80-90% of the business rules written in SQL-- yes, that bad boy. How in Poseidon's Trident do you refactor without breaking something?

The answer? Meet SQLCC. It's simple-- just do what you would normally do when faced with legacy code: write some tests and then get refactoring. To get started, write some integration tests that call a stored procedure, function, or trigger and then use SQLCC to detect the code-coverage for your tests in SQL prior to your refactor to ensure you have covered all possible code-paths. Then, start refactoring and change that good old integration test to a unit test while moving the business logic into your application code.

For now, Microsoft SQL Server is supported, but has the capability of being expanded to support additional database servers.

The Codez

In its most simple form, modify the App.config with a few values (i.e. connection string, application name, etc.) and then execute the following:

sqlcc --action=execute --target="mstest.exe" --targetArgs="/testcontainer:SQLCC.Sample.Tests.dll" --traceFileName="TraceFile"
sqlcc --action=generate --traceFileName="TraceFile"

The first line where action is set to execute, starts a MS SQL trace with the appropriate settings enabled in order to track which lines have been executed.

The second line, generates a set of HTML files with the code coverage from the specified Trace. You can also create your own OutputProvider and store the results in another file format or in the database for querying.

How Does it Work?

It works by simply running a trace with a few settings enabled in order to detect code execution paths. In order to limit traffic to just your running tests, one would simply pass in an Application Name to sqlcc and set up your tests to run using a connection string like the following:

Data Source=localhost;Initial Catalog=MY_DB;User Id=sa;Password=password;Application Name=SQLCC;

Notice "SQLCC" as the Application Name.

Is this done?

Yes, for the most part, its a really rough proof of concept (aka alpha). Definitely needs some love and unfortunately I'm not able to fully devote myself to this project.

Troubleshooting

• Ensure that "target" is pointing to the full path of your test runner. If this is MSTest, then include the full path to "mstest.exe", or if it is NUnit, include the full path to "nunit.exe".

• Ensure you have the correct "targetArgs" for your test runner. If you are using MSTest and your target is set to "C:\Path\To\mstest.exe", then try running your test runner (mstest.exe, nunit.exe, etc.) without sqlcc first with the appropriate arguments. Once you can run it correctly, copy over the test runner's arguments to "targetArgs".

• Open the App.config and modify the parameters. The important ones are "databaseConnectionString" which is your connection string to your database which you are running your tests against, "databaseApplicationName" is your Application Name as described above under "How Does it Work?", "traceFileName" is the directory which you would like to store your trace files (currently outputs *.trc files on the server)-- this is on the SQL server (must have read/write access to this directory), "outputDirectory" is the output directory for the HTML files that are generated.

• Ensure that you have copied both "SQLCC.Impl.HtmlCodeHighlighter.dll" and "SQLCC.Impl.MsSqlProvider.dll" into the same directory as "SQLCC.exe". The App.config is configured to use the providers located in these assemblies.

sqlcc's Issues

Unable to trace the DB

Hi,
I tried tracing my local DB, but I am getting the output trace file as Zero lines covered, having all the config properties set properly. Still I am getting the stored procs from the db to which I am pointing to. Kindly help me.

Mail: [email protected]

Showing correct object for objects with same name but different schema

The SQL Code Coverage Objects grid showed EFT.Hash, but text displayed was for the Cash.Hash function.

Please provide complete source code for SQLCC

Hi..The current source code available for download is incomplete. Please provide the complete source code for SQLCC. Thanks.

Screencast link in Readme

@jbarker7 Due to end of flash, can't view the video link shared in Readme. Is there an alternative?
http://screencast.com/t/5I2N4GQgcb4

first compile of SQLCC will fail on the post build since other project not built

The build order should be SQLCC last since it depends on the two other projects.

Constructor on type 'SQLCC.Impl.MsSqlProvider.MsSqlProvider' not found.

Constructor on type 'SQLCC.Impl.MsSqlProvider.MsSqlProvider' not found.

AssemblyLoader.cs: Line 68

here's the line that the exception occurs on:
return (T)Activator.CreateInstance(type, args);

it's because args = object[0] and there is no constructor for MsSqlProvider that accepts that as input.

debugging GetNamedArgumentArray shows something interesting happen on line 84:

if (providedSignature != orderedParams) continue;

Here's the local variables being compared:
orderedParams "dbp.applicationName,dbp.connString,dbp.traceDir" string
providedSignature "dbp.applicationName,dbp.connString,dbp.traceDir,dbp.traceName" string

there are no other constructors, so the foreach ends, so then the last line of the method is:

return new object[0];

This error occurs when the config file is edited (per instructions from issue: KeyNotFoundException) (config addition below)

<add key="dbp.traceName" value="TraceFile" />

I've removed the above line from the config file, and added --app.traceName to the arguments when running the EXE and all seems well. Looks like we need an update to support dbp.traceName in the config file.

KeyNotFoundException

SQLCC builds successfully, but when running, I get a KeyNotFoundException on

var startCommand = new StartCommand(outputProvider, dbProvider, arguments["app.traceName"]);

in Program.cs; line 58

It appears that another parameter "--app.traceName" is needed, but the readme doesn't explain what to put here.

Run #myTraceFile has not been completed successfully.

Run #myTraceFile has not been completed successfully.

This is all that I can get to show up in default.html after starting/stopping.

Can you please add the source code for the JS files found in:

\sqlcc-master\SQLCC.Impl.HtmlCodeHighlighter\sqlcc_output.zip

several of these JS files are "minified", and I don't see those anywhere else in this repository.

Data scrubber regex does not terminate

The finalCode = Regex.Replace... statement in DataScrubber.Scrub does not terminate.

scrubArray contains the following values:

• ([\s]*(declare[\s]+.+)+[\n\r]+([\s]*declare[\s]+.+)*[\s]*){0}
• {0}$1
• DECLARE statements are not covered since profiler does not capture them if there is no default value.

The attached file contains a cut down stored procedure illustrating the problem.

sqlcc_data_scrubber_regex_failure.txt

My workaround is to put a breakpoint on this statement and to use Set next statement to skip it.

Regex.Replace hanging

Regex.Replace is hanging here (line 22 in DataScrubber.cs) :

finalCode = Regex.Replace(finalCode, string.Format(scrubArray[0], _traceCodeFormatter.StartHighlightMarkUp, _traceCodeFormatter.EndHighlightMarkUp), string.Format(scrubArray[1], _traceCodeFormatter.StartHighlightMarkUp, _traceCodeFormatter.EndHighlightMarkUp), RegexOptions.IgnoreCase);

Parameters:
code: "-----------------------------------------------------------------------------------------\r\nCREATE FUNCTION [dbo].[YYYYMM](@datetime datetime)\r\nRETURNS CHAR(6)\r\nAS -----------------------------------------------------------------------------------------\r\nBEGIN\r\n RETURN CONVERT(CHAR(6), @datetime, 112)\r\nEND\r\n-----------------------------------------------------------------------------------------\r\n\r\n\r\n"
scrubArray[0]: "{1}([\s](--.)[\s](begin|end)[\s](--.)[\s]*){0}"
scrubArray[1]: "$1"
scrubArray[2]: "Include begin/end if it is surrounded by comments or just white-space. Also include comments."

I'm having a look as to what could be the cause, but would appreciate any assistance.

Check out these DLLs

I was implementing the SQLCC code (actually rewriting in Java) and trying to understand the code. I wasn't a fan of the regexp steps and started to look for an alternative. I ran into two .NET dlls: Microsoft.Data.Schema.ScriptDom.dll and Microsoft.Data.Schema.ScriptDom.Sql.dll. These can parse the source from a stored procedure and will tokenize it. From those tokens I build a list of statements that should get executed. Then from the trace I get the executed statements. From those two lists I calculate the statement coverage percentage. No regexp anymore. I also changed the output format to look more like Java's code coverage tool Cobertura. See this link for a rough idea of the summary page: http://www.ibm.com/developerworks/library/j-cobertura/

All in all, thanks for a great idea!!

Not able to make it run

I tried the video on this site and that usage is not working. It would be helpful, if the exact usage can be provided.
Also, if the same trace file name is provided, instead of throwing error.. is it possible to overwrite the same file

TraceFile's coverage is wrong

■my test procedure
CREATE PROCEDURE [dbo].[OutputDemo]
@name varchar(50),
@Age int,
@resName varchar(50) OUTPUT,
@reSage int OUTPUT
AS
if @Age >=100
begin
SET @reSage= @Age * 2
SET @resName = 'Name: ' + @name
end
else
begin
SET @reSage= @Age + 10
SET @resName = 'Name: ' + @name
end
■my unit test(TestProject1.dll)
public class UnitTest1 {
[TestMethod]
public void TestMethod1() {

        //var db = new PetaPoco.Database("Data Source=.;Initial Catalog=AdventureWorks2008R2;Trusted_connection=True;Application Name=SQLCC", "System.Date.SqlClient");
        var db = new PetaPoco.Database("Data Source=WIN-IVB0M9502MI\\SqlExpress;Initial Catalog=master;Persist Security Info=True;User ID=sa;Password=ko580501@_;Application Name=SQLCC", "System.Date.SqlClient");
        var opName = new SqlParameter("@resName", SqlDbType.VarChar);
        opName.Direction = ParameterDirection.Output;
        opName.Size = 50;
        var opAge = new SqlParameter("@resAge", SqlDbType.Int);
        opAge.Direction = ParameterDirection.Output;
        opAge.Size = 20;
        var s = Sql.Builder.Append(";EXEC OutputDemo @0, @1, @2 OUTPUT, @3 OUTPUT",
           "Markus299",
           299,
           opName,
           opAge
          );
        db.Execute(s);

        System.Console.WriteLine(opName.SqlValue);
        System.Console.WriteLine(opAge.SqlValue);

    }


    [TestMethod]
    public void TestMethod2() {

        //var db = new PetaPoco.Database("Data Source=.;Initial Catalog=AdventureWorks2008R2;Trusted_connection=True;Application Name=SQLCC", "System.Date.SqlClient");
        var db = new PetaPoco.Database("Data Source=SqlExpress;Initial Catalog=master;Persist Security Info=True;User ID=sa;Password=passwd;Application Name=SQLCC", "System.Date.SqlClient");
        var opName = new SqlParameter("@resName", SqlDbType.VarChar);
        opName.Direction = ParameterDirection.Output;
        opName.Size = 50;
        var opAge = new SqlParameter("@resAge", SqlDbType.Int);
        opAge.Direction = ParameterDirection.Output;
        opAge.Size = 20;
        var s = Sql.Builder.Append(";EXEC OutputDemo @0, @1, @2 OUTPUT, @3 OUTPUT",
           "Markus99",
           99,
           opName,
           opAge
          );
        db.Execute(s);

        System.Console.WriteLine(opName.SqlValue);
        System.Console.WriteLine(opAge.SqlValue);

    }
}

■run it
sqlcc --app.mode=start --action=execute --target="mstest.exe" --targetArgs="/testcontainer:TestProject1.dll" --traceFileName="TraceFile"
sqlcc --app.mode=stop --action=generate --traceFileName="TraceFile"

■result
Type Schema ObjectName FLOC LOC Characters %Covered
PROCEDURE dbo OutputDemo 16 16 301 56.06

Covere show 56.06% ,shoud be 100% I think?
my test case should be passed all code route.

Need exact syntax to use, please

Hi, Can anyone please post the exact syntax to use the exe?

I don't understand some of the things in the parameter like:

a) what will be the application name?

b) in the below command line -
sqlcc --app.mode=start --action=execute --target="mstest.exe" --targetArgs="/testcontainer:SQLCC.Sample.Tests.dll" --traceFileName="TraceF"

      1) what is target and targetArgs in the command line of the batch file?

Is my batch file going to run with the below syntax?

Please suggest and help.

Thanks,
-Sam