Tests fail with StopTest (OVERRUN) when generating a random integer (strategies.randoms) about hypothesis

[khardix]

@tybug Let me share the project so you have the complete picture, and then I'll point out the things I think are relevant.

https://git.sr.ht/~khardix/shoji, the offending tests live in tests/test__io.py. Setting HYPOTHESIS_RANDOM_BROKEN to False should trigger the bug.

Versions I'm able to trigger the bug on: hypothesis-6.108.10, shoji-0.3.1 (or current master branch: 487a74c7635650351cd3701f2a02170b5cf1dc31).

---

The random strategy is just randoms(use_true_random=False). However, it is used in conjuction with a composite strategy that generates a text-like byte string, which is then chopped to random chunks by the random strategy.
The composite strategy:

@stg.composite
def textlike_data(
    draw: stg.DrawFn,
    line_terminator: str = "\n",
    min_line_len: int = 0,
    max_line_len: int | None = _io.MAX_LINE_LEN,
) -> bytes:
    """Generate binary data that look like UTF-8 text.

    Generates the data as a list of lines,
    which makes it easier to reason about whether there is *any* line terminator preset.
    """

    alphabet = stg.characters(codec="utf-8", exclude_characters=set(line_terminator))
    line = stg.text(alphabet, min_size=min_line_len, max_size=max_line_len)
    text = stg.lists(line)
    final_terminator = stg.one_of(stg.just(""), stg.just(line_terminator))

    return bytes(
        line_terminator.join(draw(text)) + draw(final_terminator), encoding="utf-8"
    )

The textlike_data and randoms examples are then fed into a generator that simulates random chunks of data coming from a network:

def random_chunks(
    data: bytes, min_size: int = 0, *, rng: Random = RNG
) -> abc.Iterator[bytes]:
    """Split data into random-sized chunks.

    Arguments:
        data: The data to split.
        min_size: Minimal size of a chunk.

            ..note:: The last chunk may be smaller than `min_size`, even empty!

    Keyword arguments:
        rng: Random Number Generator for determining the chunk sizes.

    Yields:
        Chunks of `data`.

    Raises:
        ValueError: When `min_size` is negative.
    """

    if min_size < 0:
        raise ValueError(f"Negative minimal size: {min_size}")

    if len(data) < min_size:
        yield data
        return

    start_idx = 0
    while start_idx < len(data):
        remaining_len = len(data) - start_idx
        end_idx = start_idx + rng.randint(min(min_size, remaining_len), remaining_len)
        yield data[start_idx:end_idx]
        start_idx = end_idx

[tybug]

reproduces with:

@given(st.randoms())
def f(rng):
    rng.randint(0, 1 << 25_000)

for i in range(10):
    print(i)
    f()

this is potentially not ir-related, I haven't tried to bisect yet.

[Zac-HD]

hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/random.py

Lines 272 to 273 in 218d8a1

	elif method == "randint":
	result = self.__data.draw_integer(kwargs["a"], kwargs["b"])

I think we get the StopTest here and somehow fail to turn it into an Overrun error.

[tybug]

I think I was likely on an outdated master when I claimed I had a reproducer, because I can't reproduce on latest hypothesis anymore and my reproducer just traces back to the fix in #3991. And yet, the traceback in OP indicates this is a distinct issue (ie not caused by generate_novel_prefix).

@khardix are you able to share the offending strategy here? I can't immediately reproduce with synthetic overruns.

[tybug]

I suspect this is a bad interaction between hypothesis and pytest-trio:

@given(stg.data())
async def test_bufreciever_receive_line_reads_all_data(data):
    # force an overrun
    for _ in range(5): 
        data.draw(stg.integers(0, 1 << 25_000))

traceback

============================================================================ test session starts =============================================================================
platform darwin -- Python 3.12.3, pytest-8.3.2, pluggy-1.5.0
rootdir: /Users/tybug/Desktop/shoji
configfile: pyproject.toml
plugins: hypothesis-6.108.10, trio-0.8.0, anyio-4.4.0
collected 21 items / 20 deselected / 1 selected                                                                                                                              

tests/test__io.py F                                                                                                                                                    [100%]

================================================================================== FAILURES ==================================================================================
________________________________________________________________ test_bufreciever_receive_line_reads_all_data ________________________________________________________________
  + Exception Group Traceback (most recent call last):
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: TResult | None = func()
  |                              ^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 242, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
  |     return outcome.get_result()
  |            ^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
  |     yield
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
  |     yield
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/logging.py", line 848, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/logging.py", line 831, in _runtest_for
  |     yield
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/capture.py", line 879, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
  |     item.runtest()
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/python.py", line 1627, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/Users/tybug/Desktop/shoji/tests/test__io.py", line 67, in test_bufreciever_receive_line_reads_all_data
  |     async def test_bufreciever_receive_line_reads_all_data(data):
  |                    ^^^
  |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/core.py", line 1699, in wrapped_test
  |     raise the_error_hypothesis_found
  | BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pytest_trio/plugin.py", line 195, in _fixture_manager
    |     yield nursery_fixture
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pytest_trio/plugin.py", line 250, in run
    |     await self._func(**resolved_kwargs)
    |   File "/Users/tybug/Desktop/shoji/tests/test__io.py", line 69, in test_bufreciever_receive_line_reads_all_data
    |     data.draw(stg.integers(0, 1 << 25_000))
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/core.py", line 2115, in draw
    |     result = self.conjecture_data.draw(strategy, observe_as=f"generate:{desc}")
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2458, in draw
    |     return strategy.do_draw(self)
    |            ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/lazy.py", line 167, in do_draw
    |     return data.draw(self.wrapped_strategy)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2452, in draw
    |     return strategy.do_draw(self)
    |            ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/numbers.py", line 85, in do_draw
    |     return data.draw_integer(
    |            ^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2121, in draw_integer
    |     value = self.provider.draw_integer(
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1481, in draw_integer
    |     return self._draw_bounded_integer(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1780, in _draw_bounded_integer
    |     probe = self._cd.draw_bits(
    |             ^^^^^^^^^^^^^^^^^^^
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2612, in draw_bits
    |     self.__check_capacity(n_bytes)
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2654, in __check_capacity
    |     self.mark_overrun()
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2679, in mark_overrun
    |     self.conclude_test(Status.OVERRUN)
    |   File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2666, in conclude_test
    |     raise StopTest(self.testcounter)
    | hypothesis.errors.StopTest: 1
    +------------------------------------
--------------------------------------------------------------------------------- Hypothesis ---------------------------------------------------------------------------------
You can add @seed(102826716451383249480663158101555025900) to this test or run pytest with --hypothesis-seed=102826716451383249480663158101555025900 to reproduce this failure.
========================================================================== short test summary info ===========================================================================
FAILED tests/test__io.py::test_bufreciever_receive_line_reads_all_data - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
====================================================================== 1 failed, 20 deselected in 0.04s ======================================================================

pytest-trio 0.8.0, hypothesis 6-108.10.

pytest-trio may be swallowing StopTest and raising an exception group instead before StopTest can be excepted by __stoppable_test_function? which then bubbles up as we don't catch BaseExceptionGroup.

@Zac-HD maybe you have ideas here? 😅 I will likely be quite slow to identify the correct resolution with trio involved.

(thanks a lot for the reproducer @khardix!)

[Zac-HD]

ooooohhh, a BaseExceptionGroup containing the StopTest would indeed explain this.

In the long term I'd really like Hypothesis to "just work" with (Base)ExceptionGroups; in the short term we might end up just catching and crashing on this as a more-interpretable improvement on the status quo. We do already depend on the exceptiongroup backport, but missing the except* syntax before py311 is kinda painful...