System.ArgumentNullException : Value cannot be null about snapper

[viceice]

same as #16 but different exception.

[theramis]

Interesting.

Do you happen to have an example of a test which has this error?
I just tried to replicate it how #16 was replicated and I don't get the same error.
I get

Snapper.Exceptions.SupportedTestMethodNotFoundException : A supported test method was not found. Make sure you are using Snapper inside a supported test framework. See https://theramis.github.io/Snapper/faqs.html for more info

which is what I expected.

[WarrenFerrell]

Getting this error with netcore 2.1 Xunit for some of my snapper tests in release even when decorating the method with [Fact, MethodImpl(MethodImplOptions.NoInlining)] (but optimize false in csproj does work).

My branch that is already on netcore 3.1 doesn't have the issue so not sure how helpful this is from a debugging perspective but it's more weight to the argument to switch to [CallerMemberName] .

Message: System.ArgumentNullException : Value cannot be null. Parameter name: path1 Stack Trace: Path.Combine(String path1, String path2, String path3) SnapshotIdResolver.ResolveSnapshotId(String partialSnapshotName) Snapper.MatchSnapshot(Object snapshot, String partialSnapshotName) IntegrationApiTests.GetAssetConfigsMatchesSnapshot() --- End of stack trace from previous location where exception was thrown ---

[theramis]

Hey @WarrenFerrell,
I've been thinking about this problem lately and I think I have a nice solution.

I'm thinking of introducing the concept of passing letting a user pass in a SnapshotId which would define the location of the snapshot.
So it would end up looking something like this. (this isn't final but just an example of how it could look)

obj.ShouldMatchSnapshot(new SnapshotId(filePath, snapshotName, childSnapshotName);

What would you think about that?

The benefits of this approach are that it could work for any test framework since there would be no reliance on trying to figure out the class/method name. It lets the user completely decide where and how they want to store their snapshots.

The biggest downside for me with using [CallerMemberName] is that it only works if the ShouldMatchSnapshot method is called in the same method as the test method. If for example you had an helper method which calls ShouldMatchSnapshot and you call the helper method from the your test method. The member name would be the name of the helper method which isn't what the user would want.

[WarrenFerrell]

That's a very good point about helper methods, that would actually break some of my own tests.

For SnapshotIds, I like it There should be some validation around multiple tests calling the same snapshot though. I can easily see this being done intentionally using inlinedata but I would expect one test run to require all runtime references to the same snapshot contain the same value be the same. It would probably be preferrable if the snapshot doesn't get written when there is a conflict so the original snapshot value and the value from every reference will probably need to be stored.

[theramis]

For SnapshotIds, I like it There should be some validation around multiple tests calling the same snapshot though. I can easily see this being done intentionally using inlinedata but I would expect one test run to require all runtime references to the same snapshot contain the same value be the same. It would probably be preferrable if the snapshot doesn't get written when there is a conflict so the original snapshot value and the value from every reference will probably need to be stored.

Good point about conflicts, I didn't consider that. Passing in the snapshot does introduce some problems and limitations. I'm not sure solving for conflicts is worth it though (or even possible).
For Snapper to figure out if there is a conflict, it would need to somehow scan for all places where the SnapshotId class is being made and then figuring out is getting called by multiple tests without changing the filepath. That is super super complex and wouldn't be perfect either.
The other way would be for snapper to internally store all the snapshot ids that have been passed in during that test run, and then throw as exception if the same id was passed in twice. Unfortunately this won't be fool proof either because you can have partial test runs where not all tests are run.

The other limitation of this the snapshot id method is that in some cases the user can't use the [UpdateSnapshots] attribute. A user would generally use this snapshot id method when Snapper can't determine the test method, which case Snapper also can't check if the test method has the attribute.

I'm now thinking I could release this as an experimental feature until I/someone has a better idea.

[WarrenFerrell]

You could pass the SnapshotId via an attribute. That makes the reflective task of finding all methods using SnapshotIds vastly easier. I think it would be nice to be able to have multiple tests call the same snapshot but the initial implementation could forgo that capability as long as it doesn't restrict it from being added later.

[theramis]

You could pass the SnapshotId via an attribute. That makes the reflective task of finding all methods using SnapshotIds vastly easier. I think it would be nice to be able to have multiple tests call the same snapshot but the initial implementation could forgo that capability as long as it doesn't restrict it from being added later.

Unfortunately passing it in via attribute won't work. One of the main reasons for implementing this feature is because it's not always possible to to detect which attributes are applied onto the test method when ShouldMatchSnapshot is called.

I'll try a bunch of different things and try release this as an experimental feature soonish. It'll have a bunch of limitations but it should solve some problems for some people.

[tskimmett]

I figured I would chime in to say I'm running into this same issue with our automated test builds in Azure devops. Adding the NoInlining attribute doesn't seem to work either... Sounds like being able to hardcode the snapshot id into the test method implementation is the most guaranteed approach to get around the problem.

[WarrenFerrell]

@theramis do you know why the Null exception is thrown instead of the SupportedTestMethodNotFoundException in TestMethodResolver.cs: 41 . It seems as though the the method is found but stackFrame.GetFileName is returning null. It might be nice to have an exception when this occurs to help figure out what is going on.

[theramis]

@theramis do you know why the Null exception is thrown instead of the SupportedTestMethodNotFoundException in TestMethodResolver.cs: 41 . It seems as though the the method is found but stackFrame.GetFileName is returning null. It might be nice to have an exception when this occurs to help figure out what is going on.

Interesting haven't seen that before. I imagine it can happen when the debug files are not populated enough for it to extract the filename from. The comment here https://github.com/dotnet/runtime/blob/master/src/libraries/System.Private.CoreLib/src/System/Diagnostics/StackFrame.cs#L154-L163 suggests its extracted from the debug files.

You can try adding

<PropertyGroup Condition=" '$(Configuration)' == 'Release' ">
     <Optimize>false</Optimize>
 </PropertyGroup>

to your csproj and see if that fixes the issue.

I will update Snapper to catch this exception though and throw SupportedTestMethodNotFoundException if this use case happens.
Thanks!

[WarrenFerrell]

Sorry about the late response. Yes the Optimize tags has been the only way for me to resolve that exception. Unfortunately it is extremely difficult to reliably reproduce it. Methods that did it before no longer do.

[theramis]

I've released a fix for this in 2.2.4.