GithubHelp home page GithubHelp logo

comparethemarket / fettle Goto Github PK

View Code? Open in Web Editor NEW
66.0 66.0 10.0 20.1 MB

An experimental mutation testing tool for C# code :bug::mag:

License: MIT License

Batchfile 0.14% F# 1.17% C# 98.70%
csharp mutation-testing

fettle's People

Contributors

alexanderwinter avatar jamesbaile avatar oliwennell avatar ragsurya avatar sanderrorije avatar tiggerite avatar timheap46 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

semyondmt moellerknudsen hyabean themanfromsql dkozl robertlyson sanderrorije tiggerite alexanderwinter oliwennell

fettle's Issues

Option to run tests via command

E.g. if tests are run via a custom runner exe, allow user to specify that Fettle runs this instead of using the NUnit API. This would need to be a full command-line to include any required arguments.

This feature would be incompatible with coverage-analysis. Perhaps this should be a warning? Or perhaps required that users also add the ---skipcoverageanalysis option?

Note: this would also allow basic support for test frameworks other than NUnit. E.g. a user could specify that the XUnit test runner exe is run.

Crash on run

I try run fettle on unity3d testproject (based on nunit) and unity plugin.
fettle 0.1.0 from nuget

Analysing test coverageAn error ocurred that Fettle didn't expect.
System.AggregateException: Произошла одна или несколько ошибок. ---> System.NullReferenceException: Ссылка на объект не указывает на экземпляр объекта.
в Fettle.Core.CoverageAnalyser.InstrumentMethod(String methodId, MethodDeclarationSyntax methodNode, DocumentEditor documentEditor) в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 197
в Fettle.Core.CoverageAnalyser.d__9.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 167
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__8.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 128
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__7.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 103
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__6.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 53
--- Конец трассировки внутреннего стека исключений ---
в System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
в System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) в System.Threading.Tasks.Task1.get_Result()
в Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) в c:\dev\fettle\src\Console\Program.cs:строка 113
в Fettle.Console.Program.InternalEntryPoint(String[] args, Func3 mutationTestRunnerFactory, Func2 coverageAnalyserFactory, IOutputWriter outputWriter) в c:\dev\fettle\src\Console\Program.cs:строка 57
---> (Внутреннее исключение #0) System.NullReferenceException: Ссылка на объект не указывает на экземпляр объекта.
в Fettle.Core.CoverageAnalyser.InstrumentMethod(String methodId, MethodDeclarationSyntax methodNode, DocumentEditor documentEditor) в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 197
в Fettle.Core.CoverageAnalyser.d__9.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 167
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__8.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 128
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__7.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 103
--- Конец трассировка стека из предыдущего расположения, где возникло исключение ---
в System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
в System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
в Fettle.Core.CoverageAnalyser.d__6.MoveNext() в c:\dev\fettle\src\Core\CoverageAnalyser.cs:строка 53<---

Update documentation

  • Wiki: add to list of config file options for custom test runner
  • Wiki: new diagnostic-mode command-line argument
  • Wiki: update coverage analysis to explain incompatibility with custom test runner?
  • Readme

Run covering tests only

As an optimisation we could, after compiling mutated code, only run the tests that cover that piece of code. Right now we run all tests.

There doesn't seem to be an out-of-the-box (and free) way of telling which tests cover a given piece of code.

Rolling our own solution could involve:

  • Finding all runnable tests
  • For each test, run it and see which methods are hit
  • Recording these methods into some sort of lookup table
  • When running tests over a mutated method, using the lookup table to know which tests to run

To see which methods are hit, we could add hooks into the original source before compilation (using Roslyn) or use Microsoft's native profiler API to attach to the process and get events.

Mutators: support negating if statements

Negate the if statement itself.
E.g. if (isEnabled) mutates to if (!isEnabled)

Currently these are already supported:
if (isEnabled == true) => if (isEnabled == false)
if (isEnabled && isOtherThing) => if (isEnabled || isOtherThing)
var x = isEnabled ? 1 : 0 => var x = isEnabled ? 0 : 1

Exceptions when solution file in Fettle config has no folder in it and doesn't start with ".\"

My Fettle config file was in the same folder as my solution file 'FettleTest.sln' and the first line in my config file was 'solution: FettleTest.sln'. This causes Path.GetFullPath(directoryName) to throw an ArugmentException because the directoryName was empty. A workaround is to use '.\FettleTest.sln' in the config file.
One way to solve this would be to check whether the directory name is empty before calling Path.GetFullPath and if so, to set the directory name to ".". This causes Path.GetFullPath to work normally.
Note that when this issue as fixed like this, at least two source locations need to be fixed. Namely Filtering.RelativeDocumentPath and MutationJobList.RunAll (between the 'for' and the 'foreach' statements).
I suggest to put this duplicated code in a shared location.

Cannot compile Fettle from source using MSBuild 15.8 (Visual Studio 2017 version 15.8.0)

AppVeyor switched the "Visual Studio 2017" profile from MSBuild version 15.7 to 15.8. As soon as this happened, Fettle's CI started producing compiler warnings and test failures, failing the build.

For now, Fettle's CI settings have been modified to use the previous version and builds are green again, but this should be looked into.

Compile warnings:

Microsoft (R) Build Engine version 15.8.166+gd4e8d81a88 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [C:\projects\fettle\src\Core\Core.csproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [C:\projects\fettle\src\Console\Console.csproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "dotMemory.Unit, Version=109.0.0.0, Culture=neutral, PublicKeyToken=1010a0d8d6380325, processorArchitecture=MSIL". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\projects\fettle\src\Tests\Tests.csproj]
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved.  These reference conflicts are listed in the build log when log verbosity is set to detailed. [C:\projects\fettle\src\Tests\Tests.csproj]

Test failures:

Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants
  Expected: 1
  But was:  3
   at Fettle.Tests.Console.EndToEnd.Happy_path.<>c__DisplayClass0_0.<Console_executable_runs_and_outputs_info_about_surviving_mutants>b__0() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 45
   at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.cs:line 333
   at Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 49
1-2) Failed : Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants
  Expected: String containing "mutant(s) survived!"
  But was:  "Analysing test coverageAn error ocurred that Fettle didn't expect.
System.AggregateException: One or more errors occurred. ---> System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary`2 properties)
   at Fettle.Core.MSBuildWorkspaceFactory.Create() in C:\projects\fettle\src\Core\MSBuildWorkspaceFactory.cs:line 16
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 44
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) in C:\projects\fettle\src\Console\Program.cs:line 109
   at Fettle.Console.Program.InternalEntryPoint(String[] args, Func`3 mutationTestRunnerFactory, Func`2 coverageAnalyserFactory, IOutputWriter outputWriter) in C:\projects\fettle\src\Console\Program.cs:line 55
---> (Inner Exception #0) System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary`2 properties)
   at Fettle.Core.MSBuildWorkspaceFactory.Create() in C:\projects\fettle\src\Core\MSBuildWorkspaceFactory.cs:line 16
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 44
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
<---
"
   at Fettle.Tests.Console.EndToEnd.Happy_path.<>c__DisplayClass0_0.<Console_executable_runs_and_outputs_info_about_surviving_mutants>b__0() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 47
   at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.cs:line 333
   at Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 49
1-3) Failed : Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants
  Expected: String containing "PartiallyTestedNumberComparison.cs:7"
  But was:  "Analysing test coverageAn error ocurred that Fettle didn't expect.
System.AggregateException: One or more errors occurred. ---> System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
   at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary`2 properties)
   at Fettle.Core.MSBuildWorkspaceFactory.Create() in C:\projects\fettle\src\Core\MSBuildWorkspaceFactory.cs:line 16
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 44
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) in C:\projects\fettle\src\Console\Program.cs:line 109
   at Fettle.Console.Program.InternalEntryPoint(String[] args, Func`3 mutationTestRunnerFactory, Func`2 coverageAnalyserFactory, IOutputWriter outputWriter) in C:\projects\fettle\src\Console\Program.cs:line 55
---> (Inner Exception #0) System.IO.FileLoadException: Could not load file or assembly 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'System.Collections.Immutable, Version=1.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary`2 properties)
   at Fettle.Core.MSBuildWorkspaceFactory.Create() in C:\projects\fettle\src\Core\MSBuildWorkspaceFactory.cs:line 16
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 44
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
<---
"
   at Fettle.Tests.Console.EndToEnd.Happy_path.<>c__DisplayClass0_0.<Console_executable_runs_and_outputs_info_about_surviving_mutants>b__0() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 48
   at NUnit.Framework.Assert.Multiple(TestDelegate testDelegate) in C:\src\nunit\nunit\src\NUnitFramework\framework\Assert.cs:line 333
   at Fettle.Tests.Console.EndToEnd.Happy_path.Console_executable_runs_and_outputs_info_about_surviving_mutants() in C:\projects\fettle\src\Tests\Console\EndToEnd\Happy_path.cs:line 49
2) SetUp Error : Fettle.Tests.Core.Coverage.Happy_path
System.AggregateException : One or more errors occurred.
  ----> System.Exception : Failed to compile project HasSurvivingMutants.Implementation
warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.
error CS5001: Program does not contain a static 'Main' method suitable for an entry point
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Fettle.Tests.Core.Contexts.Coverage.When_analysing_coverage(Boolean catchExceptions) in C:\projects\fettle\src\Tests\Core\Contexts\Coverage.cs:line 104
--Exception
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 55
3) SetUp Error : Fettle.Tests.Core.Coverage.Source_file_filters
System.AggregateException : One or more errors occurred.
  ----> System.Exception : Failed to compile project HasSurvivingMutants.Implementation
warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.
error CS5001: Program does not contain a static 'Main' method suitable for an entry point
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Fettle.Tests.Core.Contexts.Coverage.When_analysing_coverage(Boolean catchExceptions) in C:\projects\fettle\src\Tests\Core\Contexts\Coverage.cs:line 104
--Exception
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 55
4) SetUp Error : Fettle.Tests.Core.Coverage.Tests_fail
System.AggregateException : One or more errors occurred.
  ----> System.Exception : Failed to compile project HasSurvivingMutants.Implementation
warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.
error CS5001: Program does not contain a static 'Main' method suitable for an entry point
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Fettle.Tests.Core.Contexts.Coverage.When_analysing_coverage(Boolean catchExceptions) in C:\projects\fettle\src\Tests\Core\Contexts\Coverage.cs:line 104
--Exception
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 55
5) Error : Fettle.Tests.Core.Coverage_analysis.Then_only_tests_that_cover_members_are_run
Moq.MockException : 
Expected invocation on the mock at least once, but was never performed: r => r.RunTests(It.IsAny<IEnumerable`1>(), It.Is<IEnumerable`1>(tn => tn.Contains<String>("example.test.one")))
Configured setups: 
x => x.RunTests(It.IsAny<IEnumerable`1>(), It.IsAny<IEnumerable`1>())
No invocations performed.
   at Moq.Mock.ThrowVerifyException(MethodCall expected, IEnumerable`1 setups, IEnumerable`1 actualCalls, Expression expression, Times times, Int32 callCount) in C:\projects\moq4\Source\Mock.cs:line 400
   at Moq.Mock.VerifyCalls(Interceptor targetInterceptor, MethodCall expected, Expression expression, Times times) in C:\projects\moq4\Source\Mock.cs:line 385
   at Moq.Mock.Verify[T,TResult](Mock`1 mock, Expression`1 expression, Times times, String failMessage) in C:\projects\moq4\Source\Mock.cs:line 321
   at Fettle.Tests.Core.Coverage_analysis.Then_only_tests_that_cover_members_are_run() in C:\projects\fettle\src\Tests\Core\Coverage_analysis.cs:line 20
6) SetUp Error : Fettle.Tests.Core.EndToEnd.Happy_path
System.AggregateException : One or more errors occurred.
  ----> System.Exception : Failed to compile project HasSurvivingMutants.Implementation
warning CS8021: No value for RuntimeMetadataVersion found. No assembly containing System.Object was found nor was a value for RuntimeMetadataVersion specified through options.
error CS5001: Program does not contain a static 'Main' method suitable for an entry point
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Fettle.Tests.Core.Contexts.EndToEnd.When_mutation_testing_the_app() in C:\projects\fettle\src\Tests\Core\Contexts\EndToEnd.cs:line 34
--Exception
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 100
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\projects\fettle\src\Core\CoverageAnalyser.cs:line 55
7) Failed : Fettle.Tests.Core.Events.Then_event_listener_is_called_when_events_occur
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Events.Then_event_listener_is_called_when_events_occur() in C:\projects\fettle\src\Tests\Core\Events.cs:line 18
8) Failed : Fettle.Tests.Core.Filtering.Projects_are_filtered.Then_only_files_matching_filter_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Projects_are_filtered.Then_only_files_matching_filter_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Projects_are_filtered.cs:line 21
9) Failed : Fettle.Tests.Core.Filtering.Projects_are_filtered_with_multiple_filters.Then_only_files_matching_either_filter_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Projects_are_filtered_with_multiple_filters.Then_only_files_matching_either_filter_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Projects_are_filtered_with_multiple_filters.cs:line 21
10) Failed : Fettle.Tests.Core.Filtering.Projects_are_not_filtered.Then_all_files_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Projects_are_not_filtered.Then_all_files_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Projects_are_not_filtered.cs:line 21
11) Failed : Fettle.Tests.Core.Filtering.Source_files_are_filtered.Then_only_files_matching_filter_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Source_files_are_filtered.Then_only_files_matching_filter_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Source_files_are_filtered.cs:line 21
12) Failed : Fettle.Tests.Core.Filtering.Source_files_are_filtered_with_multiple_filters.Then_only_files_matching_either_filter_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Source_files_are_filtered_with_multiple_filters.Then_only_files_matching_either_filter_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Source_files_are_filtered_with_multiple_filters.cs:line 23
13) Failed : Fettle.Tests.Core.Filtering.Source_files_are_not_filtered.Then_all_files_are_candidates_for_mutation
  Expected: True
  But was:  False
   at Fettle.Tests.Core.Filtering.Source_files_are_not_filtered.Then_all_files_are_candidates_for_mutation() in C:\projects\fettle\src\Tests\Core\Filtering\Source_files_are_not_filtered.cs:line 21
14) Failed : Fettle.Tests.Core.Multiple_mutants_per_syntax_node.Then_only_one_surviving_mutant_is_returned_per_syntax_node
  Expected: 1
  But was:  0
   at Fettle.Tests.Core.Multiple_mutants_per_syntax_node.Then_only_one_surviving_mutant_is_returned_per_syntax_node() in C:\projects\fettle\src\Tests\Core\Multiple_mutants_per_syntax_node.cs:line 20
15) Error : Fettle.Tests.Core.Mutants_survive.Then_the_location_of_the_surviving_mutant_is_returned
System.InvalidOperationException : Sequence contains no elements
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at Fettle.Tests.Core.Mutants_survive.Then_the_location_of_the_surviving_mutant_is_returned() in C:\projects\fettle\src\Tests\Core\Mutants_survive.cs:line 25
16) Error : Fettle.Tests.Core.Mutants_survive.Then_the_relevant_post_mutation_source_code_is_returned
System.InvalidOperationException : Sequence contains no elements
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at Fettle.Tests.Core.Mutants_survive.Then_the_relevant_post_mutation_source_code_is_returned() in C:\projects\fettle\src\Tests\Core\Mutants_survive.cs:line 39
17) Error : Fettle.Tests.Core.Mutants_survive.Then_the_relevant_pre_mutation_source_code_is_returned
System.InvalidOperationException : Sequence contains no elements
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at Fettle.Tests.Core.Mutants_survive.Then_the_relevant_pre_mutation_source_code_is_returned() in C:\projects\fettle\src\Tests\Core\Mutants_survive.cs:line 33
18) Failed : Fettle.Tests.Core.Mutants_survive.Then_the_surviving_mutant_is_returned
  Expected: 1
  But was:  0
   at Fettle.Tests.Core.Mutants_survive.Then_the_surviving_mutant_is_returned() in C:\projects\fettle\src\Tests\Core\Mutants_survive.cs:line 20

Unhandled exception when encountering expression body method

E.g.

public override string ToString() => thing;

NullReferenceException is thrown because the coverage analysis assumes all MethodDeclarationSyntax instances have a non-null Body. We should instead check for the other expression body property.

Incomplete output when mutant survives on multi-line statement

When a mutant survives on an statement that spans multiple lines, the output that's produced by Fettle will show the first line of the statement, but this doesn't necessarily show what was mutated.

For example, if the mutation of || survives in the code below:

return a == 1
   || a == 2

then the output will look something like this:

mutated from: return a == 1
mutated to: return a == 1

Give more info when errors occur during coverage analysis

When errors occur that prevent Fettle from continuing (e.g. tests fail during coverage analysis, invalid test output) then a very concise error summary is presented to the user.

However the user would benefit from more detailed info, such as callstacks and/or full NUnit output.

Error when running fake

Running fake outputs this:

The request was aborted: Could not create SSL/TLS secure channel. (Github - cached (temporarily ignore updates))

Crash when property has a setter but no getter

E.g. with the following code:

private int _thing;
public int Thing
{
   set { _thing = value; }
}

In MemberDeclarationSyntaxExtensions.CanInstrumentProperty, the call to accessors.Single(a => a.Kind() == SyntaxKind.GetAccessorDeclaration) will throw an exception

Memory usage on coverage analysis

Under certain circumstances the memory usage of Fettle increases by a very large amount during coverage analysis.

This has been noticed because of out-of-memory errors can sometimes occur, causing Fettle to exit.

We should capture a memory dump and investigate.

My suspicion is it's caused by the console output generated by the instrumented methods, in which case perhaps we can prevent a method from outputting if it's already done so for a given test.

Validate configuration

Guard against invalid configuration given in the YAML file, fail with a helpful error message.

E.g.

  • Missing required properties
  • Paths incorrect
  • Duplicate test assemblies

crash on run. 对路径“AutoMapper.dll”的访问被拒绝。

E:\Code\DevelopmentDepartment\source\OMDCCQuotes\src\OMDCCQuotes>packages\Fettle.Console.0.4.1\tools\Fettle.Console.exe -c fettle.config.yml
Fettle v0.4.1
Test coverage analysis starting...
.....................
..An error ocurred that Fettle didn't expect.
System.AggregateException: 发生一个或多个错误。 ---> System.UnauthorizedAccessException: 对路径“AutoMapper.dll”的访问被拒绝。
在 System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
在 System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
在 Fettle.Core.CoverageAnalyser.d__5.MoveNext() 位置 c:\dev\fettle\src\Core\CoverageAnalyser.cs:行号 87
--- 内部异常堆栈跟踪的结尾 ---
在 System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
在 System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) 在 System.Threading.Tasks.Task1.get_Result()
在 Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) 位置 c:\dev\fettle\src\Console\Program.cs:行号 165
在 Fettle.Console.Program.InternalEntryPoint(String[] args, Func3 mutationTestRunnerFactory, Func2 coverageAnalyserFactory, ISourceControlIntegration sourceControlIntegration, IOutputWriter outputWriter) 位置 c:\dev\fettle\src\Console\Program.cs:行号 88
---> (内部异常 #0) System.UnauthorizedAccessException: 对路径“AutoMapper.dll”的访问被拒绝。
在 System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound, WIN32_FIND_DATA& data)
在 System.IO.Directory.Delete(String fullPath, String userPath, Boolean recursive, Boolean checkHost)
在 Fettle.Core.CoverageAnalyser.d__5.MoveNext() 位置 c:\dev\fettle\src\Core\CoverageAnalyser.cs:行号 87<---

Crash when mutated code contains a void expression-bodied method

When Fettle instruments code as part of coverage analysis, it converts expression-bodied methods from, for example, this:

public int Sum(int a, int b) => a + b;

to this:

public int Sum(int a, int b) 
{
   // (instrumentation logic goes here)
   return a + b;
}

But the instrumentation code incorrectly assumes that all expression-bodied methods return a value. And so, when instrumenting something like this:

public void Log(string s) => Console.WriteLine(s);

it will wrongly convert to this, which doesn't compile:

public void Log(string s)
{
   // (instrumentation logic goes here)
   return Console.WriteLine(s);
}

Support for dotnet core

Investigate what's required.

Known issues:

  • We assume Windows style directory separators (C:\blah) - for this should be able to cherry-pick b7efed3 from the mono branch
  • MSBuildWorkspace not available

Known potential issues:

  • Relying on app-domains for isolation when running tests using NUnit Engine, not sure what happens when using the .NET Standard version?

Embedded resources are not included when Fettle compiles assemblies

This affects coverage analysis and mutation testing.

When Fettle instruments or mutates assemblies they should still contain any embedded resources that the original had. Tests may rely on these existing, so Fettle can produce false positives or fail coverage analysis.

Support null-coalesce operator ??

Fettle doesn't currently support ??, as in a = b ?? c.

Following discussions on Gitter (https://gitter.im/fettle-mutation-testing/Lobby), the proposal is to mutate the code as is, rather than expanding out ?? into its simpler underlying form and then mutating that. I.e the proposal is to mutate a = b ?? c to a = c ?? b

In the alternative, a = b ?? c could be re-written as if (b != null) { a = b; } else { a = c; } which could then be mutated using existing rules for the !=.

However, this has at least two problems:

  1. It makes it harder to report surviving mutants, because the source code of the surviving mutant will be too distant from the original source code;
  2. This could have to be done recursively, as the statement might be a = b ?? c ?? d ?? e.

The current proposal provides an answer to the two problems above, but does raise a general problem of which this is a specific instance.

The general problem is to do with re-writing or other magic done behind the scenes by the compiler or run-time. In this case it is the syntactic sugar to do with ??. With this proposal, mutating and re-writing are not commutative. We're accepting this in the interests of getting something done that's simple and useful, rather than aiming for theoretical purity.

If you start with a = b ?? c you could mutate then re-write, or re-write then mutate. With this proposal they don't lead to the same result.

  1. a = b ?? c mutates to a = c ?? b and then re-writes to if (c != null) {a = c;} else {a = b;}
  2. a = b ?? c re-writes to if (b != null) {a = b;} else {a = c;} and then mutates to if (b == null) {a = b; } else {a = c;}

Exclude list for sources to mutate

This is not an issue, but a wish for an exclude filter:

So far it is possible to include source files and folder and that works very well, but we typically have our tests in a subfolder named 'Test' within the source project. Of course we could create sourceFileFilters for all files and folders that contain sources and leave the 'Test' folder out, but if we add additional sources and sourcefolders, we do have to adapt the config file every time.

So it would be much nicer to be able to simply exclude the 'Test' folder.

Configuration validation is done too late

The configuration file is validated just before performing mutation testing (top of MutationTestRunner.Run() method).

This validation happens after coverage analysis takes place. So any errors in configuration that affect coverage analysis will not result in a friendly error message but a crash.

Crash when mutated code contains partial method declaration

This line in CoverageAnalyser.cs:

var firstChildNode = methodNode.Body.ChildNodes().FirstOrDefault();

...throws a NullReferenceException when .Body is null, causing Fettle to quit.

.Body will be null when a method doesn't have a "normal" body. Fettle already handles this for expression-bodied methods and abstract methods, but not partial method declarations. E.g.

public abstract void Thing();

Not handling tests that use TestCaseSource correctly

Saw some code in the wild that was being tested via a TestCaseSource and produced surviving mutants, but when the test was refactored to use a foreach instead (i.e. testing the same things but via a normal [Test]), the mutants no longer survived.

Crash due to unexpected NUnit test suite result

It seems that Fettle doesn't expect test suits to complete with a status of "Skipped" (expects "Passed" or "Failed"). A skipped status seems quite possible, so should be supported.

Perhaps an easier way to implement the logic (in NUnitRunResult.Parse) would be to look at the number of test failures (//test-run/failures) instead of the overall result (//test-run/result) which can be misleading? I.e. I've seen a result of "Skipped" even though some tests still ran.

Stacktrace:

...An error ocurred that Fettle didn't expect.
System.AggregateException: One or more errors occurred. ---> System.InvalidOperationException: Unexpected NUnit test run result: "Skipped"
   at Fettle.Core.Internal.NUnit.NUnitRunResults.Parse(XmlNode rootNode) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitRunResults.cs:line 33
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunTests(IEnumerable`1 testAssemblyFilePaths, IEnumerable`1 testMethodNames, ITestEventListener testEventListener) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 66
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunTests(IEnumerable`1 testAssemblyFilePaths, IEnumerable`1 testMethodNames) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 25
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunAllTests(IEnumerable`1 testAssemblyFilePaths) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 30
   at Fettle.Core.Internal.MutationJob.<Run>d__16.MoveNext() in C:\dev\fettle\src\Core\Internal\MutationJob.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.Internal.MutationJobList.<RunAll>d__2.MoveNext() in C:\dev\fettle\src\Core\Internal\MutationJobList.cs:line 57
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.MutationTestRunner.<Run>d__5.MoveNext() in C:\dev\fettle\src\Core\MutationTestRunner.cs:line 47
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Fettle.Console.Program.InternalEntryPoint(String[] args, Func`3 mutationTestRunnerFactory, Func`2 coverageAnalyserFactory, IOutputWriter outputWriter) in C:\dev\fettle\src\Console\Program.cs:line 72
---> (Inner Exception #0) System.InvalidOperationException: Unexpected NUnit test run result: "Skipped"
   at Fettle.Core.Internal.NUnit.NUnitRunResults.Parse(XmlNode rootNode) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitRunResults.cs:line 33
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunTests(IEnumerable`1 testAssemblyFilePaths, IEnumerable`1 testMethodNames, ITestEventListener testEventListener) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 66
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunTests(IEnumerable`1 testAssemblyFilePaths, IEnumerable`1 testMethodNames) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 25
   at Fettle.Core.Internal.NUnit.NUnitTestEngine.RunAllTests(IEnumerable`1 testAssemblyFilePaths) in C:\dev\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:line 30
   at Fettle.Core.Internal.MutationJob.<Run>d__16.MoveNext() in C:\dev\fettle\src\Core\Internal\MutationJob.cs:line 75
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.Internal.MutationJobList.<RunAll>d__2.MoveNext() in C:\dev\fettle\src\Core\Internal\MutationJobList.cs:line 57
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.MutationTestRunner.<Run>d__5.MoveNext() in C:\dev\fettle\src\Core\MutationTestRunner.cs:line 47<---

Add list of not mutated methods to verbose output

If a method is OK for test coverage but has no contents that can be mutated, it is silently omitted from all output, even verbose. Having a list of methods (and possibly properties) that couldn't be mutated would help users not have a false sense of security / nasty surprises when they discover that mutation isn't checking everything they thought it was.

Report all errors when loading .sln

Currently we ignore some of the errors that may occur when a .sln (solution) file is read by Roslyn.

We need to check the Diagnostics property of the workspace and throw an exception/return an error to the user if it contains any items after loading the solution.

Documentation

We should have:

  • Code of conduct ✅
  • License ✅
  • Contributor guide ✅
  • Issue template
  • A gitter channel ✅

Other things:

  • Developer getting started (need VS 2017, ...) ✅
  • How it all works (technical details)
  • Known issues - ✅

Crash on running with fettle.config.example.yml

I have built successfully from the source code (also HasSurvivingMutants project), but when I tried running .\src\Console\bin\Debug\Fettle.Console.exe -c .\fettle.config.example.yml it doesn't work. The error messages are shown below. Anyone has ideas? Thanks ahead!

An error ocurred that Fettle didn't expect.
System.AggregateException: One or more errors occurred. ---> System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeAssembly.get_DefinedTypes()
at System.Composition.Hosting.ContainerConfiguration.<>c.b__16_0(Assembly a)
at System.Linq.Enumerable.d__172.MoveNext() at System.Composition.TypedParts.TypedPartExportDescriptorProvider..ctor(IEnumerable1 types, AttributedModelProvider attributeContext)
at System.Composition.Hosting.ContainerConfiguration.CreateContainer()
at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable1 assemblies) at Microsoft.CodeAnalysis.Host.Mef.DesktopMefHostServices.get_DefaultServices() at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary2 properties)
at Fettle.Core.MSBuildWorkspaceFactory.Create() in fettle\src\Core\Internal\MSBuildWorkspaceFactory.cs:line 16
at Fettle.Core.Config.d__33.MoveNext() in fettle\src\Core\Config.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Fettle.Core.Config.<HasAnyMutatableDocuments>d__34.MoveNext() in fettle\src\Core\Config.cs:line 73 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task1.get_Result() at Fettle.Console.Program.Run(String[] args, ITestRunnerFactory testRunnerFactory, Func4 mutationTestRunnerFactory, Func2 coverageAnalyserFactory, ISourceControlIntegration sourceControlIntegration, IOutputWriter outputWriter) in fettle\src\Console\Program.cs:line 100 ---> (Inner Exception #0) System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information. at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module) at System.Reflection.RuntimeAssembly.get_DefinedTypes() at System.Composition.Hosting.ContainerConfiguration.<>c.<WithAssemblies>b__16_0(Assembly a) at System.Linq.Enumerable.<SelectManyIterator>d__172.MoveNext()
at System.Composition.TypedParts.TypedPartExportDescriptorProvider..ctor(IEnumerable1 types, AttributedModelProvider attributeContext) at System.Composition.Hosting.ContainerConfiguration.CreateContainer() at Microsoft.CodeAnalysis.Host.Mef.MefHostServices.Create(IEnumerable1 assemblies)
at Microsoft.CodeAnalysis.Host.Mef.DesktopMefHostServices.get_DefaultServices()
at Microsoft.CodeAnalysis.MSBuild.MSBuildWorkspace.Create(IDictionary2 properties) at Fettle.Core.MSBuildWorkspaceFactory.Create() in fettle\src\Core\Internal\MSBuildWorkspaceFactory.cs:line 16 at Fettle.Core.Config.<FindMutatableDocuments>d__33.MoveNext() in fettle\src\Core\Config.cs:line 66 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Fettle.Core.Config.d__34.MoveNext() in fettle\src\Core\Config.cs:line 73<---

Handle tests hanging

Looks like we wait indefinitely for tests to complete after mutation.
But they could hang and/or mutation could introduce a hang.

So we should time out after some period.
When timed out should this result in a different kind of result which is neither pass or fail? E.g. "timed out" instead of "no mutant survived" or "a mutant survived"?

Code within less common methods and properties is not mutated

Constructors are not included in coverage analysis or mutation testing by Fettle.

Code that needs to change:

  1. MemberDeclarationSyntaxExtensions.CanInstrument method:
if (memberDeclaration is MethodDeclarationSyntax methodDeclaration)
{
    return CanInstrumentMethod(methodDeclaration);
}
else if (memberDeclaration is PropertyDeclarationSyntax propertyDeclaration)
{
    return CanInstrumentProperty(propertyDeclaration);
}

Need to handle memberDeclaration being of type ConstructorDeclarationSyntax.

  1. SyntaxNodeExtensions.NameOfContainingMember method:
if (node is NamespaceDeclarationSyntax @namespace) foundNamespace = @namespace;
else if (node is ClassDeclarationSyntax @class) foundClass = @class;           
else if (node is MethodDeclarationSyntax method) foundMethod = method;
else if (node is PropertyDeclarationSyntax property) foundProperty = property;

Need to handle node being of type ConstructorDeclarationSyntax

Re-implement filtering

The current filtering behaviour is affected by an apparent bug in the glob library.

To get around this we could:

  • Use regular expressions instead (and use the built-in regex API in .NET)
  • Remove support for filtering of source files and require exact match on project names

Test fixtures marked as explicit are still run

As part of coverage analysis and probably during mutation testing too.

If the explicit attribute is on a method, then it is correctly ignored.
But if it's on a class, then it is not ignored and the test(s) are run.

This is caused by the NUnitExploreResults.Parse method only looks at test-case elements in isolation. Instead, it should discard test-cases whose ancestor test-suites' runstate is not runnable.

Make coverage/test impact analysis optional

E.g. via a command line switch.

Default behaviour is to only mutate covered code. If users aren't keeping an eye on their coverage metrics, they may never spot code that is completely uncovered. Running fettle over their code base could give them a false sense of security.

Disabling the analysis would also allow a work-around for the OOM issue (#20).

Only mutate modified source files

This is Raghav's idea
Work out which files changed in the most recent commit, and only mutate those.

This feels like it could make running Fettle an "every CI build" thing.

Structs with mutatable constructors are not supported

There is a core dump if you try to use a struct that has a mutatable constructor such as

public struct MyStruct
{
    int minAllowed, maxAllowed;

    public MyStruct(int min, int max)
    {
        if (min > max) {throw new Exception("oh dear");}

        minAllowed = min;
        maxAllowed = max;
    }
}

The core dump happens in NUnitCoverageCollector::onTestComplete. The member in memberIdsToNames that matches the key has a null value.

The cause of the null value is SyntaxNodeExtensions::NameOfContainingMember. It starts with the parameter list for the struct's constructor, goes up a level and correctly finds a match for MemberDeclarationSyntax (because ConstructorDeclarationSyntax = grand-child of MemberDeclarationSyntax). It goes up another level and fails. It finds StructDeclarationSyntax rather than ClassDeclarationSyntax and they are peers rather than parent/child, so there is no match. As the class hasn't been found, the method returns null, which is fed into memberIdsToNames.

Crash when auto property exists within subclass

If this sublcass:

public class Thingy
{
      public bool IsThing => true;
 }

...is added to the PartiallyTestedNumberComparison class in the examples, then when coverage analysis is performed a crash occurs within Roslyn:

System.AggregateException: One or more errors occurred. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.CSharpExtensions.Kind(SyntaxNode node)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.GetDeclarationKind(SyntaxNode declaration)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.AsNodeLike(SyntaxNode existingNode, SyntaxNode newNode)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.ReplaceNode(SyntaxNode root, SyntaxNode declaration, SyntaxNode newDeclaration)
   at Microsoft.CodeAnalysis.Editing.SyntaxEditor.ReplaceChange.Apply(SyntaxNode root, SyntaxGenerator generator)
   at Microsoft.CodeAnalysis.Editing.SyntaxEditor.GetChangedRoot()
   at Microsoft.CodeAnalysis.Editing.DocumentEditor.GetChangedDocument()
   at Fettle.Core.Internal.Instrumentation.<InstrumentDocument>d__1.MoveNext() in C:\dev\fettle\src\Core\Internal\Instrumentation.cs:line 45
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 121
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 97
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 48
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) in C:\dev\fettle\src\Console\Program.cs:line 123
   at Fettle.Console.Program.InternalEntryPoint(String[] args, Func`3 mutationTestRunnerFactory, Func`2 coverageAnalyserFactory, IOutputWriter outputWriter) in C:\dev\fettle\src\Console\Program.cs:line 71
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.CSharpExtensions.Kind(SyntaxNode node)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.GetDeclarationKind(SyntaxNode declaration)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.AsNodeLike(SyntaxNode existingNode, SyntaxNode newNode)
   at Microsoft.CodeAnalysis.CSharp.CodeGeneration.CSharpSyntaxGenerator.ReplaceNode(SyntaxNode root, SyntaxNode declaration, SyntaxNode newDeclaration)
   at Microsoft.CodeAnalysis.Editing.SyntaxEditor.ReplaceChange.Apply(SyntaxNode root, SyntaxGenerator generator)
   at Microsoft.CodeAnalysis.Editing.SyntaxEditor.GetChangedRoot()
   at Microsoft.CodeAnalysis.Editing.DocumentEditor.GetChangedDocument()
   at Fettle.Core.Internal.Instrumentation.<InstrumentDocument>d__1.MoveNext() in C:\dev\fettle\src\Core\Internal\Instrumentation.cs:line 45
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileProject>d__7.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 121
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Fettle.Core.CoverageAnalyser.<InstrumentThenCompileMultipleProjects>d__6.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 97
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Fettle.Core.CoverageAnalyser.<AnalyseCoverage>d__5.MoveNext() in C:\dev\fettle\src\Core\CoverageAnalyser.cs:line 48<---

Option to display diagnostic output

When trying to understand behaviour it would be useful to have more info about what Fettle is doing.

The idea is that a user would enable diagnostics logging via a command-line option (like with quiet mode). Outputting to the console directly (instead of a file) should be fine initially.

Things to log might include:

  • The mutations being applied
  • Result of test execution
  • ...

Support mutation of code within properties

Coverage analysis and mutation testing analysis both iterate over methods, looking for syntax nodes to mutate.

This ignores, for example, expressions within properties or field initialisation.

E.g.

public class Thing
{
   private int a =  5 * 100; // is not mutated
   private int b;

   public int C1 => a + b; // is not mutated
   public int C2 { get { return a + b; } } // is not mutated

   public int D()
   {
      return a + b; // is mutated
   }
}

Note that Fettle does support expression-bodied methods (support added in #18).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.