Random failures with hypothesis.errors.StopTest exception raised about hypothesis

[tomato42]

(yeah, yeah, I know some of those words.jpg)

Unfortunately I'm not able to help with this. What I can say, that I have some other non-trivial usages of hypothesis, ones that are complex enough that sometimes they time-out with old versions; so they may be good to verify sanity of complex test generation.

[tybug]

Thanks for the report! I have no doubt this regressed in #3818. I'll take a look later today.

[tybug]

Sorry, I know this is a pretty frustrating bug to run into. I'll look into this within the week.

[tybug]

Hmm. This is an overrun during generate_novel_prefix, due to the possibility that drawing a single node during novel generation takes more than BUFFER_SIZE in extreme cases. In the case above, this is a combination of unlucky successive probes + large max integer bounds (1500 bytes per probe).

One solution is retrying these overruns in generate_novel_prefix via a try/catch of StopTest. I'm nervous this will end in infinite loops or severe slowdowns in extreme cases, but I can't think of something that would repeatedly trigger this behavior in generate_novel_prefix while avoiding HealthCheck.too_large, so maybe this would be ok? (thinking aloud: what about a case where every novel prefix is > BUFFER_SIZE, so we hit this infinite loop before reaching enough overruns to raise the health check?)

[tomato42]

@tybug I still have regular failures because of this. Any chance of this getting fixed soon?

[khardix]

I also run into this recently. While I cannot offer any asistance in fixing it, I have a workaround: using flaky and marking the test as being flaky, re-running it if it fails with StopTest exception. So far re-running the test once is usually enough for me to not trigger this bug.

[tomato42]

with such level of workarounding, I'd rather just mark it as skipped, and point to this bug