nunit / nunit.analyzers Goto Github PK

Roslyn analyzers for writing unit tests with NUnit

License: MIT License

C# 99.76% PowerShell 0.22% Batchfile 0.01% Shell 0.02%

nunit nunit-analyzers roslyn-analyzer dotnet csharp hacktoberfest

nunit.analyzers's Introduction

NUnit Analyzers

This is a suite of analyzers that target the NUnit testing framework. Right now, the code is separate from the NUnit framework, so if you want to try out the analyzers you'll need to download the analyzers separately as a nuget package. In the future the analyzers may be added as part of the NUnit framework package but that hasn't been done yet.

Download

The latest stable release of the NUnit Analyzers is available on NuGet or can be downloaded from GitHub. Note that for Visual Studio 2017 one must use versions below 3.0 - note that these versions are no longer updated, so version 2.10.0 is the last version that works in Visual Studio 2017. Version 3.0 and upwards require Visual Studio 2019 (version 16.3) or newer, these versions also enables supression of compiler errors such as errors arising from nullable reference types.

Prerelease nuget packages can be found on MyGet. Please try out the package and report bugs and feature requests.

Analyzers

The full list of analyzers can be found in the documentation.

Below we give two examples of analyzers. One will look for methods with the [TestCase] attribute and makes sure the argument values are correct for the types of the method parameters along with the ExpectedResult value if it is provided.

The other analyzer looks for classic model assertions (e.g. Assert.AreEqual(), Assert.IsTrue(), etc.). This analyzer contains a fixer that can translate the classic model assertions into constraint model assertions (Assert.That()).

Which version works with Unity Test Framework

If your Unity project is made by Unity under 2021.2, then use NUnit.Analyzers v2.x.

If your Unity project is made by Unity 2021.2 or later, then use NUnit.Analyzers v3.3 (v3.4 or later of the analyzers does not work with Unity).

You should use an analyzer built with the same version of Microsoft.CodeAnalysis.CSharp as the one embedded in the Unity Editor.

License

NUnit analyzers are Open Source software and released under the MIT license, which allow the use of the analyzers in free and commercial applications and libraries without restrictions.

Contributing

There are several ways to contribute to this project. One can try things out, report bugs, propose improvements and new functionality, work on issues (especially the issues marked with the labels help wanted and Good First Issue), and in general join in the conversations. See Contributing for more information.

This project has adopted the Code of Conduct from the Contributor Covenant, version 1.4, available at http://contributor-covenant.org/version/1/4. See the Code of Conduct for more information.

Contributors

NUnit.Analyzers was created by Jason Bock. A complete list of contributors can be found on the GitHub contributors page.

nunit.analyzers's People

Contributors

Stargazers

Watchers

nunit.analyzers's Issues

NRE in ClassicModelAssertUsageAnalyzer.AnalyzeInvocation

The method ClassicModelAssertUsageAnalyzer.AnalyzeInvocation can fail with a NRE, since invocationSymbol can be null. The problem is that the type of context.SemanticModel.GetSymbolInfo(invocationNode.Expression).Symbol is SourceSimpleParameterSymbol and not IMethodSymbol. The problem can be seen by opening the file nunit.nunit\src\NUnitFramework\tests\Constraints\MsgUtilTests.cs (in the nunit-project). Then the code fails in

 [Test]
 public static void FormatValue_ContextualCustomFormatterInvoked_FactoryArg()
 {
     TestContext.AddFormatter(next => val => (val is CustomFormattableType) ? "custom_formatted" : next(val));

     Assert.That(MsgUtils.FormatValue(new CustomFormattableType()), Is.EqualTo("custom_formatted"));
 }

and invocationNode is referencing next(val). I have not look much into the problem yet.

Analyzer for TestCaseAttribute should also work for nullable types

The analyzer should also handle the following (from nunit\src\NUnitFramework\tests\Attributes\TestCaseAttributeTests.cs). Currently, it gives a warning The value of the argument at position 0 cannot be assigned to the argument x.

        [TestCase(1)]
        public void CanConvertIntToNullableLong(long? x)
        {
            Assert.That(x.HasValue);
            Assert.That(x.Value, Is.EqualTo(1));
        }

NRE when writing an attribute before adding a reference to NUnit

The following code makes the analyzer throw NRE if one has not added a reference to NUnit.

[TestFixture]
public class Class1
{
    [Test]
    public void Test1()
    {
    }
}

The method should probably just return if testCaseType is null.

Exception

System.NullReferenceException was unhandled by user code
  HResult=-2147467261
  Message=Object reference not set to an instance of an object.
  Source=nunit.analyzers
  StackTrace:
       at NUnit.Analyzers.TestCaseUsage.TestCaseUsageAnalyzer.AnalyzeAttribute(SyntaxNodeAnalysisContext context) in F:\TEMP\Github\nunit.analyzers\src\nunit.analyzers\TestCaseUsage\TestCaseUsageAnalyzer.cs:line 53
       at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.<>c__DisplayClass42_1`1.<ExecuteSyntaxNodeAction>b__1()
       at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock(DiagnosticAnalyzer analyzer, Action analyze, Nullable`1 info)
  InnerException:

Analyzer for TestCaseAttribute should also work for generic TestFixtures

The analyzer should also handle the following (from \nunit\src\NUnitFramework\tests\Internal\TypeParameterUsedWithTestMethod.cs). Currently, it gives a warning The value of the argument at position 0 cannot be assigned to the argument x.

    [Category("Generics")]
    [TestFixture(typeof(double))]
    public class TypeParameterUsedWithTestMethod<T>
    {
        [TestCase(5)]
        [TestCase(1.23)]
        public void TestMyArgType(T x)
        {
            Assert.That(x, Is.TypeOf(typeof(T)));
        }
    }

Add error if tests/testcases have a return value, but no ExpectedResult

If tests/testcases have a return value, but no ExpectedResult then it fail with the message
"Message: Method has non-void return value, but no result is expected". We should handle this.

[Test]
public string Test3()
{
    return "12";
}

[TestCase(1)]
public string Test4(int i)
{
    return "12";
}

Add CI support for master and PRs

Probably using appveyor and cake. To begin with we will first focus on the build and testing part. A later issue will add package functionality.

TestCase analyser doesn't handle nullables

The TestCase analyser incorrectly flags up the below test, with nullable parameters.

I had a quick test with optional parameters and params, and they both look good!

Suggestion: Analyzer Should Fix TestCaseSource(string) usage to use nameof()

Thank you for this tool.

We have several Test Cases that look similar to this:

[TestCaseSource( "_SetupIsNotEqualToNoEntryErrorBehaviorTestCases" )]

These were written before the introduction of nameof() and if they were to be re-written today would look like this:

[TestCaseSource( nameof(_SetupIsNotEqualToNoEntryErrorBehaviorTestCases) )]

This way Intellisense would operate correctly (F12/Renaming/Find Usages/etc).

I realize that you would be required to use a compiler/IDE that supports C# 6.0 (that usually means Visual Studio 2015) but it would encourage others to keep current.

Thank you for your consideration.

Analyzer: warn on Console.WriteLine in tests.

Happens that things are dumped when debugging tests and then forgotten. Reasons to remove:

They slow down tests.
Pollute CI logs.

Remove dependency on NUnit for analyzer project.

Better to use strings here.

Warning if literal or const value is provided as actual value

In most cases (actually can't figure out when it should actually be otherwise) const value shouldn't be provided as actual value to assert.
However it can often happen when actual and expected parameter are mixed places.
That mostly applies to classic assertion model, but might be helpful for constraint model as well.

Add editorconfig

Depending on #20

Message: Async test method must have Task<T> return type when a result is expected

We should warn about the following problem.

Message: Async test method must have Task<T> return type when a result is expected

    [TestCase(ExpectedResult = 1)]
    public async System.Threading.Tasks.Task AsyncTaskTestCaseWithExpectedResult()
    {
      await Task.Run(() => 1);
    }

and

    [TestCase(ExpectedResult = 1)]
    public System.Threading.Tasks.Task TaskTestCaseWithExpectedResult()
    {
      return Task.Run(() => 1);
    }

NRE in AnalyzePositionalArgumentsAndParameters

In the method TestCaseUsageAnalyzer.AnalyzePositionalArgumentsAndParameters there is the assumption that all expressions in the attribute has type LiteralExpressionSyntax. This is not always the case. The assumption fails in e.g. nunit.nunit\src\NUnitFramework\tests\Constraints\MsgUtilTests.cs, where we have the constant s52 which hence has type IdentifierNameSyntax.

        private const string s52 = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";

        [TestCase(s52, 52, 0, s52, TestName="NoClippingNeeded")]
        [TestCase(s52, 29, 0, "abcdefghijklmnopqrstuvwxyz...", TestName="ClipAtEnd")]
        [TestCase(s52, 29, 26, "...ABCDEFGHIJKLMNOPQRSTUVWXYZ", TestName="ClipAtStart")]
        [TestCase(s52, 28, 26, "...ABCDEFGHIJKLMNOPQRSTUV...", TestName="ClipAtStartAndEnd")]
        public static void TestClipString(string input, int max, int start, string result)
        {
            System.Console.WriteLine("input=  \"{0}\"", input);
            System.Console.WriteLine("result= \"{0}\"", result);
            Assert.That(MsgUtils.ClipString(input, max, start), Is.EqualTo(result));
        }

Do Not Allow Any Assertions to be Used in TearDown

Description

Based on discussion from here: nunit/docs#149
If a method is marked with [TearDown], no assertion calls from the Assert class should be allowed.

Example

[TearDown]
public void TearDown()
{
  Assert.That(true, Is.True); // This line would be flagged.
}

Analyzer Message

"No methods from the Assert class may be used in a [TearDown] method"

Default Severity Level

Error

Code Fixes

none

"Sequence contains no elements" in ClassicModelAssertUsageAnalyzer.AnalyzeInvocation

The code has the assumption that every InvocationExpression is inside a MethodDeclarationSyntax. This is not the case for nunit.nunit\src\NUnitFramework\tests\Api\FrameworkControllerTests.cs that has invocations as part of of a field assignment

public class FrameworkControllerTests
{
...
    private static readonly string MOCK_ASSEMBLY_NAME = typeof(MockAssembly).GetTypeInfo().Assembly.FullName;
...
}

So the code fails with an System.InvalidOperationException - "Sequence contains no elements" - in

if(!context.Node.Ancestors().OfType<MethodDeclarationSyntax>().Single().ContainsDiagnostics)

Examine concurrent execution af analyzers

Links for inspiration

Add CONTRIBUTING.md

Add document describing the contribution guidelines.

ClassicModelAssertUsageAnalyser throws InvalidOperationException

I haven't looked in to this too much, but I believe it may be when it encounters a property in the TestFixture class.

Error	AD0001	Analyzer 'NUnit.Analyzers.ClassicModelAssertUsage.ClassicModelAssertUsageAnalyzer' threw an exception of type 'System.InvalidOperationException' with message 'Sequence contains no elements'.	XXX		1	Active	Analyzer 'NUnit.Analyzers.ClassicModelAssertUsage.ClassicModelAssertUsageAnalyzer' threw the following exception:
'Exception occurred with following context:
Compilation: XXX
SyntaxTree: XXX
SyntaxNode: XXX[InvocationExpressionSyntax]@[6922..6945) (162,25)-(162,48)

System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at NUnit.Analyzers.ClassicModelAssertUsage.ClassicModelAssertUsageAnalyzer.AnalyzeInvocation(SyntaxNodeAnalysisContext context)
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.<>c__DisplayClass42_1`1.<ExecuteSyntaxNodeAction>b__1()
   at Microsoft.CodeAnalysis.Diagnostics.AnalyzerExecutor.ExecuteAndCatchIfThrows_NoLock(DiagnosticAnalyzer analyzer, Action analyze, Nullable`1 info)
-----
'.

Add Roslyn Code Analyzers for NUnit

@rprouse commented on Mon May 04 2015

Visual Studio 2015 and Roslyn introduced code analyzers and the concept of code-aware libraries. There are many ways that users can incorrectly use NUnit, so we could provide analyzers to detect incorrect usage and suggest fixes. This would help users become more productive and reduce the number of errors reported because features are incorrectly used.

If we do this, it should be a separate analyzer assembly that is a dependency of the framework NuGet package.

Some things we could check,

Constructors and test methods are public
TestCase attributes have the correct number and type of arguments for the given method
If a test method has a return type, the test attribute has ExpectedResult set
If we Assert.That(x, Is.Not.Null) and variants, x is a reference type
Classes that contain tests are marked with the TestFixture attribute
Usage of TestCaseSource within the same class (see #320)

Good idea? Are there other improper usages that cannot be caught at compile time that we should consider?

@CharliePoole commented on Mon May 04 2015

I like the idea. The key question is when do we do it - before or after 3.0. Clearly any time we spend on it is taken away from other features, so how important is it?

Moving ahead, I think it could be a separate project under the nunit organization.

Currently, we detect most of the errors you describe at runtime. I think we should continue to do that, marking the tests as non-runnable. It has always been a principle of NUnit that we treat anything you try to mark as a test as a test, giving an error if it's not quite right. That's definitely better than silently ignoring the errors at runtime.

Comments on the errors listed:

Constructors and test methods are public
That makes sense. The issue is what's a test method and what's a test fixture. We don't want to fall into the trap of some third-party runners, maintaining separate code for identifying tests. Ideally, we would use NUnit itself to do the identification, but that requires an assembly.
TestCase attributes have the correct number and type of arguments for the given method
Yes
If a test method has a return type, the test attribute has ExpectedResult set
Yes
If we Assert.That(x, Is.Not.Null) and variants, x is a reference type
Should be a warning, unless NUnit is modified to give an error. Currently, such a test "works" - that is, Assert.That(5, Is.Not.Null) succeeds.
Classes that contain tests are marked with the TestFixture attribute
It's optional for most kinds of tests, so that one doesn't make sense to me.
Usage of TestCaseSource within the same class (see #320)
Definitely! Should be an error if there is no default constructor, otherwise a warning. When we deprecate certain patterns that can't be marked with [Obsolete] this would be the way to do it.

Others:

Test Fixtures with no tests
Test methods with no asserts
TestFixture on abstract methods

Maybe we should set up a spec page for this in the dev wiki.

@rprouse commented on Tue May 05 2015

I think that this is a post 3.0 release task, I just wanted to document it. I agree that it should probably be in a separate repository and with your other comments. I will update the labels and milestone and we can come back to this later.

@rprouse commented on Tue May 26 2015

Another candidate would be testing the correct usage of the Range attribute. See #472

@CharliePoole commented on Sun Jul 03 2016

@rprouse This issue came up in a discussion with @ChrisMaddock who is interested in working on it. I can assign it to him if you have no objection, but I think we ought to clarify some stuff first.

From reading back, it seems that your vision was of something we would distribute to users. Would this be a standalone program? Or would it plug into some existing static analyzer package as an extension? If that's the case, would we expect to support more than one analyzer? Or is all this stuff - as I expect - what we need to figure out?

I guess this ended up on our Backlog when I eliminated the Future milestone last year. Based on how we categorize things nowadays, this seems like an "idea" trying to become a "feature", and I relabeled it accordingly. Since @ChrisMaddock has started experimenting with this, he would have the job of shepherding the idea on the way to feature-hood, but I think having a bit more about what you imagined as a way to delivering this feature would help him.

Of course, this is likely to end up as an entirely separate project from NUnit, but right now this seems like the best place to discuss it.

@ChrisMaddock commented on Sun Jul 03 2016

Thanks for writing this up @CharliePoole. :-)

Presuming Rob's talking about what I think he is - Roslyn analyzers are essentially 'plug-ins' to Roslyn, that can provide users with warnings/code refactorings in Visual Studio - inline, and prior to compilation. Hence why NUnit would be a great fit - working via reflection means we currently can't detect a lot of user errors until runtime. Distribution is by nuget package, or vsix. Wrapping up framework and analyzer in a single package seems to be the suggested method when already distributing via nuget package - although we'd need to decide (I imagine much later on!) if this is in the default framework packages, or additional ones.

An additional area this could be really useful is in NUnit2 -> 3 updates - I've lost count of the number of times I've find/replace'd IsStringContaining()! Using static analyzers, this could be a one click operation to adjust across the whole solution.

I'll try and get a couple of the simpler examples up soon, and we can see what we're looking at.

@rprouse commented on Sun Jul 03 2016

@ChrisMaddock You are thinking what I am thinking 👍

Feel free to grab the issue and run with it. I think we will want these in a separate repository unless we plan to ship them in the Framework NuGet package. When they were introduced at Build a couple of years ago, the idea was that they would be bundled with libraries. It is a cool idea to do it that way, but I also don't like forcing them on people. Then again, they just provide code fixes.

For now, maybe start in a personal repository and we can discuss where they belong?

@ChrisMaddock commented on Sun Jul 03 2016

Sounds like a plan, will try a few. :-)

@rprouse commented on Sat Nov 19 2016

@JasonBock, this is the issue that contains other ideas for the Roslyn analyzer. I will assign it to you once you accept the team invitation. We might want to move it to the new repo once you start working there and convert it to a ZenHub epic or split it into child issues.

@JasonBock commented on Fri Nov 25 2016

I'm really close to getting a first commit done, should be done tomorrow :)

One quick comment on one idea @CharliePoole had mentioned before....

"Test methods with no asserts" - It's not uncommon for me to write unit tests where I'm using a mocking framework and I don't have any Assert assertions, but my assertions are in the calls that verify the mock was used correctly - e.g. myMockObject.VerifyAll(). So I'm not sure this is something that can be enforced all the time.

@godrose commented on Fri Nov 25 2016

It looks like all failure points can be reduced to 2 cases: assertions
which are performed upon the SUT and verifications that are performed upon
a mock.
Can the latter be mapped with a regex? Possibly with specific statements
for each isolation framework.

On 25 Nov 2016 07:38, "Jason Bock" [email protected] wrote:

I'm really close to getting a first commit done, should be done tomorrow :)

One quick comment on one idea @CharliePoole
https://github.com/CharliePoole had mentioned before....

"Test methods with no asserts" - It's not uncommon for me to write unit
tests where I'm using a mocking framework and I don't have any Assert
assertions, but my assertions are in the calls that verify the mock was
used correctly - e.g. myMockObject.VerifyAll(). So I'm not sure this is
something that can be enforced all the time.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
nunit/nunit#629 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHbkIzWxc5HJcNgdAJvhjC9RI16KbBFSks5rBnQ8gaJpZM4EPgXP
.

@JasonBock commented on Fri Nov 25 2016

@godrose it could be, if we're willing to keep track of all mocking frameworks and how they work, which wouldn't be trivial. Also, what if a team decides to use some helper methods to do assertions and then you wouldn't know that the test is "correct" because it is doing assertions, just indirectly. Or what if the developer is using another assertion framework like FluentAssertions? I don't think an analyzer can be written that would cover all the cases effectively - the scope is just too broad.

@godrose commented on Fri Nov 25 2016

Certainly.
You don't have to cover all cases. Just the most important ones. Like you
would in a product case. You don't solve all use cases when you first
release a product to the market, you only hit the ones that maximize the
value for the client.

As to usage inference I absolutely agree with you. I don't use NUnit
assertions at all preferring FluentAssertions instead. Helper methods are
out of question unless you want to compile the delegates/reflected info to
expressions and so on.

Just consider the cost and the value for one particular case and see if
it's worth a shot.

On 25 Nov 2016 16:59, "Jason Bock" [email protected] wrote:

@godrose https://github.com/godrose it could be, if we're willing to keep
track of all mocking frameworks and how they work, which wouldn't be
trivial. Also, what if a team decides to use some helper methods to do
assertions and then you wouldn't know that the test is "correct" because it
is doing assertions, just indirectly. Or what if the developer is using
another assertion framework like FluentAssertions? I don't think an
analyzer can be written that would cover all the cases effectively - the
scope is just too broad.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
nunit/nunit#629 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHbkI9AOIjdLEpos9s-0STLNZHyuApkwks5rBvfSgaJpZM4EPgXP
.

@JasonBock commented on Fri Nov 25 2016

@godrose, agreed, and that's kind of what I'm getting at. There is no solid use case for this particular analyzer request. It's just too broad, and I think time can be used more wisely addressing other cases.

@godrose commented on Fri Nov 25 2016

Fair enough. Good luck in developing the project!

On 25 Nov 2016 18:44, "Jason Bock" [email protected] wrote:

@godrose https://github.com/godrose, agreed, and that's kind of what
I'm getting at. There is no solid use case for this particular analyzer
request. It's just too broad, and I think time can be used more wisely
addressing other cases.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
nunit/nunit#629 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHbkIyN1YsxnBcqGVhe6_hOwBrDyRBL6ks5rBxBbgaJpZM4EPgXP
.

@CharliePoole commented on Fri Nov 25 2016

@JasonBock I may have the wrong idea about what you are implementing. My assumption was that the user or team would get to choose which checks to apply using some sort of settings dialog or file. If that's the case then a simple check for asserts is feasible, but if there are no choices available to the user then only things that are always wrong can be flagged.

I guess it will be clearer when there's code to look at.

@JasonBock commented on Fri Nov 25 2016

@CharliePoole VS allows the user to turn analyzers on and off and change their setting (error, warning, info) from the default if they want. But there's no way (or no easy way) to change what the analyzer does with configuration. And yes, once I get the initial code in everyone can try them and give feedback. I WILL have something in soon, promise!

@JasonBock commented on Fri Nov 25 2016

Finally pushed the code:

https://github.com/nunit/nunit.analyzers

So....I think the next step is for team members and others to review it and provide feedback/issues/etc. I can address those and fix any issues, and then I think getting stories and tasks in ZenHub in place for the rest of the analyzers so they can be tracked easier, rather than lumping them all into one issue here.

@CharliePoole commented on Sat Nov 26 2016

I'd like to exercise this but I could use some instructions. The readme takes me through building and installing, but not usage.

nunit / nunit.analyzers Goto Github PK

nunit.analyzers's Introduction

NUnit Analyzers

Download

Analyzers

Which version works with Unity Test Framework

License

Contributing

Contributors

nunit.analyzers's People

Contributors

Stargazers

Watchers

Forkers

nunit.analyzers's Issues

Description

Example

Analyzer Message

Default Severity Level

Code Fixes

Description:

Example:

Analyzer Message:

Code Fixes:

Description:

Example:

Analyzer Message:

Default Severity Level:

Code Fixes:

Recommend Projects

Recommend Topics

Recommend Org

Jobs