Investigate: .NET Framework support about stryker-net

[rouke-broersma]

@oliwennell Thanks for the suggestion! We'll see if we can use this to fix our issues locating dependencies in GAC.

[kevinlyles]

I played around with this a little bit already. At a minimum:

• The target framework version for .csproj files targeting .NET Framework seems to live in a TargetFrameworkVersion node instead of TargetFramework like .NET Core.
• The output path should be retrieved from the project file instead of assumed -- .NET Framework seems to have a different default than .NET Core, but we should respect the configured path anyway.

However, there's probably more -- I seem to get fewer mutations than I'd expect on the toy .NET Framework project I was trying this with.

[richardwerkman]

@kevinlyles Did you try the toy project with Stryker.NET? If so, it is correct that it could mutate the project? That would be great news. What framework version was the project?

About the fewer mutations, that could be because of the lack of mutators at the moment?

We'll keep in mind that a new project file reader has to be created for the .NET framework.

[kevinlyles]

Yep, I had to make the changes I made above, but it did seem to mutate some of the code.

I know there aren't many mutators yet, but it seemed like it was missing things that even the existing mutators should have caught. I may have missed something, though.

[simondel]

For an initial setup we could try running the DDL's using vstest.exe. We can use the Visual Studio Test task for TFS/VSTS as a reference on how to pass specific arguments.

I would suggest manually running the command on your machine first to see what kind of output you get from vstest.exe if all tests pass or if a test fails. The performance will probably be quite bad if we simply run all tests using vstest.exe (we won't be able to use incremental test runs because the DDL's don't change).

Simply checking for the exit code of the command would be fine to begin with. I'd prefer to have basic support over having no support at all. We may be able to determine the vstest.exe path based on the environment variables. The TFS/VSTS task is also able to do this, since it has a Visual Studio version 'Latest'.

I don't think we'll have any issues with mutating the code but it's good to also try that out in the early stage (we can simply disable/remove the initial test run to test this)

[richardwerkman]

Today at the Stryker hackathon we've had a discussion about how to support .NET Framework. Our conclusion was that the two biggest changes we would have to make are:

• The testrunner should also run .NET Framework tests.
  • The VSTest adapter can run tests both on .NET Framework and .NET Core.
• Stryker would have to be executed differently.
  • The "dotnet" command does not exist in .NET Framework. The CLI currently uses the extensibility model. For .NET Framework an executable should be called another way through the command line.

An possible sollution for resolving the VSTest executable location can be found here cake-build/cake#2077

We have looked at fettle for inspiration about how to execute Stryker as a .NET Framework executable. We now think installing using the package manager console is a good idea. After installing the executable can be executed through this console.

[simondel]

How about keeping the current CLI for .NeT Core and making a new one for .NET Framework? That fixes multiple issues at once.

[richardwerkman]

@simondel Creating seperate CLI projects won't fix the issues we are having. And we would lose a lot of reusable code. Right now we are experimenting with a CLI that targets multiple runtimes.

[richardwerkman]

Full Framework doesn't work with transient dependencies. This breaks resolving referenced assemblies. One workaround could be to move the msbuild task to the "project under test".

Another workaround could be to write a custom resolver for full framework. This resolver can read the .csproj file and analyse the xml.

[rouke-broersma]

Useful articles for building NuGet Tool Installer compatible with both dotnet core and dotnet framework

http://www.marcusoft.net/2011/12/creating-tools-only-nuget-package.html
https://natemcmaster.com/blog/2017/11/11/build-tools-in-nuget/

[yaelkeemink]

I can now find all the csproj files and load all its references. When I try to compile this code right now I get the following errormessage: Predefined type 'System.String' is not defined or imported
This means it did not load the mscorlib. I cant seem to find why it did not load, is there anyone who knows the solution to this? I found another issue on the roslyn github page: dotnet/roslyn#12393

The code that tries to compile is as follows:

                syntaxTrees: syntaxTrees,
                options: new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary),
                references: _input.AssemblyReferences);

            // first try compiling
            var emitResult = compiler.Emit(ms);```

[richardwerkman]

Maybe it has to do with the way you resolved the AssemblyReferences? I don't know about your implementation but it could be that it always misses mscorlib (and only mscorlib). In that case we could add it hardcoded on .net framework from the default location. I guess it's fair to assume that every .net framework project needs mscorlib as a referenced assembly.

[simondel]

@yaelkeemink Is working on this on the _.NET_Framework_support branch

[oliwennell]

I think I came across something similar re: the System.String / mscorlib issue.

What worked for me in the end was using the Microsoft.Build.Locator nuget package (github repo).

E.g.

internal static class MSBuildWorkspaceFactory
{
    static MSBuildWorkspaceFactory()
    {
        MSBuildLocator.RegisterDefaults();
    }

    public static MSBuildWorkspace Create()
    {
        return MSBuildWorkspace.Create(
            new Dictionary<string, string>
            {
                { "CheckForSystemRuntimeDependency", "true" }
            });
    }
}

and then

using (var workspace = MSBuildWorkspaceFactory.Create())
{
   var solution = await workspace.OpenSolutionAsync(SolutionFilePath);
   // change code then compile solution...
}

I've seen online that CheckForSystemRuntimeDependency can also help so added it just in case ¯_(ツ)_/¯