Sporadic results (XCTest) when running simultaneous tests about nocilla

[subbarao]

@bredfield did you find fix for your issues. I am also experiencing similar issues.

[bredfield]

@subbarao I haven't been able to get stable results with Nocilla. I've been using Mockingjay, which has worked well.

[paudav]

Hi @bredfield,

TL;DR: This is probably caused by the exception thrown here. When this gets thrown, subsequent requests in the same test run to correctly stubbed URLs seem to fail.

I'm working in Objective-C, but I've run into a similar issue. Since you've changed test suites, this probably isn't a big deal for you, but I'll leave this post here for anybody else who is seeing inconsistent results during test runs.

In my case, our class under test is using NSURLConnection with delegate callbacks directed to an NSOperationQueue.

As described in issue #103, the exception thrown at https://github.com/luisobo/Nocilla/blob/master/Nocilla/LSNocilla.m#L79 basically wrecks all of the tests that run after the exception is thrown.

In Xcode, the tests seem to run in alphabetical order of the test method names--Depending on the order of tests your XCTest class implementation, this could make things appear to be "randomly" failing. (For fun, reorder the test methods in your file alphabetically, and see if a more clear failure pattern appears).

In our case, the test that caused the exception was intentionally hitting a non-stubbed URL in order to test behavior with non-responsive requests. By design, this test succeeded after checking some queuing behaviors related to a bunch of open requests (to non-stubbed URLs). Inside Nocilla, of course, this throws an exception that we never see, and subsequent tests with correctly stubbed requests fail.

The result, of course, is that if you run each of the tests individually, they pass. If you run them all together, the non-stubbed request in the middle of the test run breaks every subsequent test.

You can detect this condition by setting an exception breakpoint. If you try this and it breaks into LSNocilla.m during the test run, you've sent a request to a non-stubbed URL.

Good Luck,
Paul B. Davis

[TheAdamBorek]

Hi guys,
has anyone resolved this issue ?

[TheAdamBorek]

As @paudav mention above it was connected with exception raising. There is an pull request. Error is returned instead of raising an exception