Busy loop randomly runs 6x slower causing flaky DeadlineExceeded errors about hypothesis

[jobh]

And thanks for the detailed investigation! It supplied all the clues 😁

[jobh]

After the first failure, the test is re-run under tracing to try to give more information about the failure. The busy loop is probably close to worst case for tracing performance impact.

To verify, you can add

import sys
sys.settrace(lambda *args: None)

at the top of the test. This will effectively disable hypothesis' tracing, since only one tracer can be attached.

Note that tracing is much faster in python >= 3.12.

[mscheifer]

After the first failure, the test is re-run under tracing to try to give more information about the failure. The busy loop is probably close to worst case for tracing performance impact.

To verify, you can add

import sys
sys.settrace(lambda *args: None)

at the top of the test. This will effectively disable hypothesis' tracing, since only one tracer can be attached.

Note that tracing is much faster in python >= 3.12.

Ahh yeah this is it. That does indeed make the issue go away. As does doing:

@settings(phases=[p for p in Phase if p != Phase.explain])

[Zac-HD]

This is an unfortunate experience, but I don't think we can really improve on it - any solution would also make the logic more complicated, and therefore make Hypothesis harder to understand. Only a few years until all supported Python versions are fast, though...

[mscheifer]

Is it worth adding something to the error message? Like this message
Unreliable test timings! On an initial run, this test took 239.58ms, which exceeded the deadline of 100.00ms, but on a subsequent run it took 32.53 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
could be extended to add
If you're testing CPU-bound pure-Python code, consider disabling the explain phase as it can slow down runs (or upgrade to Python 3.12).
for Python < 3.12

Or add a note in the docs? Maybe here https://hypothesis.readthedocs.io/en/latest/settings.html#controlling-what-runs