False positive SupportedTestMethodNotFoundException with xUnit about snapper

[theramis]

Thanks for making an issue!

Hmm that is a very interesting case. I didn't consider this initially and I'll have to look into it further. Nothing I can think of the top of my head on how to fix this.

Have you looked into using CallerMemberName instead of StackTrace (this should be much faster and should not need removing inline optimization)?

I did look into this initially but it doesn't work well if the ShouldMatchSnapshot() method is not called from the main test method. So I opted to use stacktrace to figure out the Xunit test name.

Or how about you could specify name of snapshot file in some kind of annotation?

This has been on my mind lately and atm is the only real solution I can think of for this issue.

I'm currently out of town for a few days for work so won't be able to look at this. Will try have a look 🤞 in the weekend.

[tomasbruckner]

I was just doing a little research and I don't think CallerMemberName will work, because it doesn't work with extensions. Or rather it works but it returns name of the extension method. There is a proposal but I don't think it will be implemented anytime soon https://developercommunity.visualstudio.com/idea/566788/new-attribute-similar-to-callermembername-but-for.html

The most quickiest way would be to pass the name of the snapshot as a parameter to shouldMatchSnapshot.

[tomasbruckner]

I have done some digging and in my opinioin the best would be to use CallerMemberName, which is determinated in the compile time, which means it is much faster than stacktrace (in some benchmarks 25 times faster). It works fine with the async example above.

This will have two caveats, the first one it no longer can be extension method. The second one is that it is true that it will no longer work if it is not called in the main test method. To solve this, I would suggest not saving one file per method, but use one file per class. And basically use ShouldMatchChildSnapshot functionality. So for one class, we would have JSON like this:

For class MyTestClass.cs with two test method Test1 - with 2 matchSnapshot calls and Test2 - with 1 matchSnapshot calls:

{
  "Test1 1": {},
  "Test1 2": {},
  "Test2 1": {},
}

And you could unite functionality of ShouldMatchSnapshot with ShouldMatchChildSnapshot and make just one method with optional name. This would be pretty similar how Jest do it (Jest also have option to specify property matchers. This would be also pretty handy but not relevant to this issue).

[theramis]

There are quite a few points mentioned in the above comment so I'll try address them one by one. 😄

I have done some digging and in my opinioin the best would be to use CallerMemberName, which is determinated in the compile time, which means it is much faster than stacktrace (in some benchmarks 25 times faster). It works fine with the async example above.

I don't think using CallerMemberName would be best. It would be a breaking change and that would mean Snapper v3 needs to be made. It also means that Snapper would not work with test frameworks which generate code. e.g. Specflow. There is no way to add code into the file where the Xunit test method is in that case.

You also mentioned speed above, is that currently a problem? or would it just be a happy benefit of using CallerMemberName. I know relative to stacktrace, CallerMemberName is super fast but in normal operation stacktrace shouldn't be considered "slow" in my opinion.

This will have two caveats, the first one it no longer can be extension method. The second one is that it is true that it will no longer work if it is not called in the main test method. To solve this, I would suggest not saving one file per method, but use one file per class. And basically use ShouldMatchChildSnapshot functionality. So for one class, we would have JSON like this:

For class MyTestClass.cs with two test method Test1 - with 2 matchSnapshot calls and Test2 - with 1 matchSnapshot calls:

{
  "Test1 1": {},
  "Test1 2": {},
  "Test2 1": {},
}

And you could unite functionality of ShouldMatchSnapshot with ShouldMatchChildSnapshot and make just one method with optional name. This would be pretty similar how Jest do it (Jest also have option to specify property matchers. This would be also pretty handy but not relevant to this issue).

I'm not sure how that would work. You mentioned "The second one is that it is true that it will no longer work if it is not called in the main test method.". How would making one snapshot file solve the issue of knowing which class/method/file the method ShouldMatchSnapshot was called from? You would still need to call ShouldMatchSnapshot from the method with the [Fact] attribute so that the correct class/method is used right?

Also in case you didn't know Snapper does currently support storing snapshots for one class in one file like Jest. see https://theramis.github.io/Snapper/snapper/snapshots-per-class.html

[theramis]

My current thinking is that there are a couple of options which could be done while being backwards compatible.

1. Add a new method which can take in the method type, name or something like that as part of ShouldMatchSnapshot. Something like this perhaps ShouldMatchSnapshot(classname, methodname).
2. Make Snapper extensible like in v1. I could expose some of the interfaces for people to use. That way depending on the use case you could implement your own way to detect the test method. Whether its CallerMemberName, hardcoded or whatever.
   a. e.g. expose SnapshotIdResolver as ISnapshotIdResolver which people could implement
   b. e.g. expose ITestMethodResolver which people could implement.
3. Similar to option 1 but expose SnapshotId which people could pass in which would avoid the framework having to determine the test method.

The second option is probably the end goal from my point of view but requires a lot more thinking before I can expose those interfaces.

Did you have any thoughts @tomasbruckner?

[theramis]

I've released an experimental feature in 2.2.4. See https://theramis.github.io/Snapper/#/pages/snapper/advanced_snapshot_control. This feature should allow you to create a snapshot in exceptional use cases like this.