Crash on run about fettle

[nsobiratel]

thank you for answer. i will try search bug reason. possible rework tests

[oliwennell]

Thanks for the report @nsobiratel , I will look into this soon.

[oliwennell]

From looking at the stack trace, it seems that Fettle is expecting a method declaration to have a body, but it doesn't. I'm thinking this might be a partial method, which Fettle doesn't handle correctly yet.

Does the code you're running Fettle against contain partial methods @nsobiratel ?

[oliwennell]

I've raised #25 for not handling empty partial method declarations

[nsobiratel]

project contains two partial class, but no partial methods

[nsobiratel]

build from github crash with another error

Analysing test coverageAn error ocurred that Fettle didn't expect.
System.AggregateException: Произошла одна или несколько ошибок. ---> NUnit.Engine.NUnitEngineException: The NUnit 3 driver encountered an error while executing reflected code. ---> System.InvalidCastException
: Не удалось привести прозрачный прокси к типу "System.Web.UI.ICallbackEventHandler".
в NUnit.Framework.Api.FrameworkController.LoadTestsAction..ctor(FrameworkController controller, Object handler)
--- Конец трассировки внутреннего стека исключений ---
в NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
в NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary2 settings) в NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) в NUnit.Engine.Runners.DirectTestRunner.LoadPackage() в NUnit.Engine.Runners.TestDomainRunner.LoadPackage() в NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded() в NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) в NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter) в Fettle.Core.Internal.NUnit.NUnitTestEngine.FindTests(IEnumerable1 testAssemblyFilePaths) в C:\Users\at\Documents\deb2dev\reportunit\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:строка 15
в Fettle.Core.CoverageAnalyser.d__6.MoveNext() в C:\Users\at\Documents\deb2dev\reportunit\fettle\src\Core\CoverageAnalyser.cs:строка 67
--- Конец трассировки внутреннего стека исключений ---
в System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
в System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) в System.Threading.Tasks.Task1.get_Result()
в Fettle.Console.Program.AnalyseCoverage(ICoverageAnalyser coverageAnalyser, IOutputWriter outputWriter, Config config) в C:\Users\at\Documents\deb2dev\reportunit\fettle\src\Console\Program.cs:строка 111
в Fettle.Console.Program.InternalEntryPoint(String[] args, Func3 mutationTestRunnerFactory, Func2 coverageAnalyserFactory, IOutputWriter outputWriter) в C:\Users\at\Documents\deb2dev\reportunit\fettle\sr
c\Console\Program.cs:строка 55
---> (Внутреннее исключение #0) NUnit.Engine.NUnitEngineException: The NUnit 3 driver encountered an error while executing reflected code. ---> System.InvalidCastException: Не удалось привести прозрачный прок
си к типу "System.Web.UI.ICallbackEventHandler".
в NUnit.Framework.Api.FrameworkController.LoadTestsAction..ctor(FrameworkController controller, Object handler)
--- Конец трассировки внутреннего стека исключений ---
в NUnit.Engine.Drivers.NUnit3FrameworkDriver.CreateObject(String typeName, Object[] args)
в NUnit.Engine.Drivers.NUnit3FrameworkDriver.Load(String testAssemblyPath, IDictionary2 settings) в NUnit.Engine.Runners.DirectTestRunner.LoadDriver(IFrameworkDriver driver, String testFile, TestPackage subPackage) в NUnit.Engine.Runners.DirectTestRunner.LoadPackage() в NUnit.Engine.Runners.TestDomainRunner.LoadPackage() в NUnit.Engine.Runners.DirectTestRunner.EnsurePackageIsLoaded() в NUnit.Engine.Runners.DirectTestRunner.Explore(TestFilter filter) в NUnit.Engine.Runners.MasterTestRunner.Explore(TestFilter filter) в Fettle.Core.Internal.NUnit.NUnitTestEngine.FindTests(IEnumerable1 testAssemblyFilePaths) в C:\Users\at\Documents\deb2dev\reportunit\fettle\src\Core\Internal\NUnit\NUnitTestEngine.cs:строка 15
в Fettle.Core.CoverageAnalyser.d__6.MoveNext() в C:\Users\at\Documents\deb2dev\reportunit\fettle\src\Core\CoverageAnalyser.cs:строка 67<---

[nsobiratel]

i think trouble in second case in specific type of unity3d tests (they have returned type IEnumerator, attribute UnityTest instead Test, but UnityTest inherited from Test) and i cannot run fettle in the future :( or exists variant delegate run tests to unity (it can start from command line)?

[oliwennell]

It seems that at least one other person has encountered this error when trying to run NUnit against Unity3d tests. See last comments on these issues:
nunit/nunit#2719
nunit/nunit-console#345

(Fettle and the NUnit console both share the NUnit API code which seems to be throwing the exception).

The links above seem to suggest that a work-around may be to separate the "pure .NET" (POCO) code/tests from those that reference Unity3d.

Unfortunately I don't know Unity3d enough to say if there's a proper fix that can be made, so I'm gonna have to say that right now Fettle doesn't support Unity3d :(

If you manage to find any more info feel free to let me know.

[NikolayShitov]

Hi @oliwennell, i'm nsobiratel :) I do big work with our tests. Test project seperated on two part - one part run on device (Windows, Android, iOS, Mac, Uwp, Webgl) and generate data. Second part - console nunit application (self hosted custom runner) change some testing settings, do little environment setup and run tests - they validate generated data and receive some command line args. And now I want try use fettle. But could't - in config only assemblies, i cannot use custom runner. We can adopt fettle for my use case?

[oliwennell]

Hi @NikolayShitov 👋

Thanks for the suggestion!

Right now Fettle works by using the NUnit API to run tests individually. This allows it to do optimisations, such as only running tests that are relevant to code being mutated or stopping test execution as soon as any test has failed.

If Fettle was to run the tests via an exe instead, it would not be able to apply these optimisations and run much more slowly. However, I can see how it could be useful to do this in some cases, so will create an issue to add it as a command-line option.

If you'd rather not wait, you should be able to set Fettle's testAssemblies configuration property to look at your custom test runner exe. However it would only work correctly if you changed your tests so that:

• The custom runner assembly contained all the tests that it ran.
• All setup was done via NUnit initialisation methods (e.g. [OneTimeSetUp] or [SetUpFixture] methods)

[oliwennell]

Created #52 for this

[oliwennell]

Version 0.5.0 introduced support for custom test runners.

If this doesn't work for you feel free to raise another issue @NikolayShitov