Comments (6)
@tybug Let me share the project so you have the complete picture, and then I'll point out the things I think are relevant.
https://git.sr.ht/~khardix/shoji, the offending tests live in tests/test__io.py
. Setting HYPOTHESIS_RANDOM_BROKEN
to False should trigger the bug.
Versions I'm able to trigger the bug on: hypothesis-6.108.10, shoji-0.3.1 (or current master branch: 487a74c7635650351cd3701f2a02170b5cf1dc31
).
The random strategy is just randoms(use_true_random=False)
. However, it is used in conjuction with a composite strategy that generates a text-like byte string, which is then chopped to random chunks by the random
strategy.
The composite strategy:
@stg.composite
def textlike_data(
draw: stg.DrawFn,
line_terminator: str = "\n",
min_line_len: int = 0,
max_line_len: int | None = _io.MAX_LINE_LEN,
) -> bytes:
"""Generate binary data that look like UTF-8 text.
Generates the data as a list of lines,
which makes it easier to reason about whether there is *any* line terminator preset.
"""
alphabet = stg.characters(codec="utf-8", exclude_characters=set(line_terminator))
line = stg.text(alphabet, min_size=min_line_len, max_size=max_line_len)
text = stg.lists(line)
final_terminator = stg.one_of(stg.just(""), stg.just(line_terminator))
return bytes(
line_terminator.join(draw(text)) + draw(final_terminator), encoding="utf-8"
)
The textlike_data
and randoms
examples are then fed into a generator that simulates random chunks of data coming from a network:
def random_chunks(
data: bytes, min_size: int = 0, *, rng: Random = RNG
) -> abc.Iterator[bytes]:
"""Split data into random-sized chunks.
Arguments:
data: The data to split.
min_size: Minimal size of a chunk.
..note:: The last chunk may be smaller than `min_size`, even empty!
Keyword arguments:
rng: Random Number Generator for determining the chunk sizes.
Yields:
Chunks of `data`.
Raises:
ValueError: When `min_size` is negative.
"""
if min_size < 0:
raise ValueError(f"Negative minimal size: {min_size}")
if len(data) < min_size:
yield data
return
start_idx = 0
while start_idx < len(data):
remaining_len = len(data) - start_idx
end_idx = start_idx + rng.randint(min(min_size, remaining_len), remaining_len)
yield data[start_idx:end_idx]
start_idx = end_idx
from hypothesis.
reproduces with:
@given(st.randoms())
def f(rng):
rng.randint(0, 1 << 25_000)
for i in range(10):
print(i)
f()
this is potentially not ir-related, I haven't tried to bisect yet.
from hypothesis.
hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/random.py
Lines 272 to 273 in 218d8a1
I think we get the StopTest
here and somehow fail to turn it into an Overrun
error.
from hypothesis.
I think I was likely on an outdated master when I claimed I had a reproducer, because I can't reproduce on latest hypothesis anymore and my reproducer just traces back to the fix in #3991. And yet, the traceback in OP indicates this is a distinct issue (ie not caused by generate_novel_prefix
).
@khardix are you able to share the offending strategy here? I can't immediately reproduce with synthetic overruns.
from hypothesis.
I suspect this is a bad interaction between hypothesis and pytest-trio:
@given(stg.data())
async def test_bufreciever_receive_line_reads_all_data(data):
# force an overrun
for _ in range(5):
data.draw(stg.integers(0, 1 << 25_000))
traceback
============================================================================ test session starts =============================================================================
platform darwin -- Python 3.12.3, pytest-8.3.2, pluggy-1.5.0
rootdir: /Users/tybug/Desktop/shoji
configfile: pyproject.toml
plugins: hypothesis-6.108.10, trio-0.8.0, anyio-4.4.0
collected 21 items / 20 deselected / 1 selected
tests/test__io.py F [100%]
================================================================================== FAILURES ==================================================================================
________________________________________________________________ test_bufreciever_receive_line_reads_all_data ________________________________________________________________
+ Exception Group Traceback (most recent call last):
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
| result: TResult | None = func()
| ^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 242, in <lambda>
| lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
| return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
| return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
| return outcome.get_result()
| ^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
| raise exc.with_traceback(exc.__traceback__)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
| teardown.throw(outcome._exception)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
| yield from thread_exception_runtest_hook()
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
| yield
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
| teardown.throw(outcome._exception)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
| yield from unraisable_exception_runtest_hook()
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
| yield
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
| teardown.throw(outcome._exception)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/logging.py", line 848, in pytest_runtest_call
| yield from self._runtest_for(item, "call")
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/logging.py", line 831, in _runtest_for
| yield
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
| teardown.throw(outcome._exception)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/capture.py", line 879, in pytest_runtest_call
| return (yield)
| ^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 167, in _multicall
| teardown.throw(outcome._exception)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
| return (yield)
| ^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
| res = hook_impl.function(*args)
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
| item.runtest()
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/python.py", line 1627, in runtest
| self.ihook.pytest_pyfunc_call(pyfuncitem=self)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
| return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
| return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
| raise exception.with_traceback(exception.__traceback__)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
| res = hook_impl.function(*args)
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
| result = testfunction(**testargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/tests/test__io.py", line 67, in test_bufreciever_receive_line_reads_all_data
| async def test_bufreciever_receive_line_reads_all_data(data):
| ^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/core.py", line 1699, in wrapped_test
| raise the_error_hypothesis_found
| BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pytest_trio/plugin.py", line 195, in _fixture_manager
| yield nursery_fixture
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/pytest_trio/plugin.py", line 250, in run
| await self._func(**resolved_kwargs)
| File "/Users/tybug/Desktop/shoji/tests/test__io.py", line 69, in test_bufreciever_receive_line_reads_all_data
| data.draw(stg.integers(0, 1 << 25_000))
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/core.py", line 2115, in draw
| result = self.conjecture_data.draw(strategy, observe_as=f"generate:{desc}")
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2458, in draw
| return strategy.do_draw(self)
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/lazy.py", line 167, in do_draw
| return data.draw(self.wrapped_strategy)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2452, in draw
| return strategy.do_draw(self)
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/strategies/_internal/numbers.py", line 85, in do_draw
| return data.draw_integer(
| ^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2121, in draw_integer
| value = self.provider.draw_integer(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1481, in draw_integer
| return self._draw_bounded_integer(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 1780, in _draw_bounded_integer
| probe = self._cd.draw_bits(
| ^^^^^^^^^^^^^^^^^^^
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2612, in draw_bits
| self.__check_capacity(n_bytes)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2654, in __check_capacity
| self.mark_overrun()
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2679, in mark_overrun
| self.conclude_test(Status.OVERRUN)
| File "/Users/tybug/Desktop/shoji/myenv/lib/python3.12/site-packages/hypothesis/internal/conjecture/data.py", line 2666, in conclude_test
| raise StopTest(self.testcounter)
| hypothesis.errors.StopTest: 1
+------------------------------------
--------------------------------------------------------------------------------- Hypothesis ---------------------------------------------------------------------------------
You can add @seed(102826716451383249480663158101555025900) to this test or run pytest with --hypothesis-seed=102826716451383249480663158101555025900 to reproduce this failure.
========================================================================== short test summary info ===========================================================================
FAILED tests/test__io.py::test_bufreciever_receive_line_reads_all_data - BaseExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
====================================================================== 1 failed, 20 deselected in 0.04s ======================================================================
pytest-trio 0.8.0, hypothesis 6-108.10.
pytest-trio may be swallowing StopTest
and raising an exception group instead before StopTest
can be excepted by __stoppable_test_function
? which then bubbles up as we don't catch BaseExceptionGroup
.
@Zac-HD maybe you have ideas here? 😅 I will likely be quite slow to identify the correct resolution with trio involved.
(thanks a lot for the reproducer @khardix!)
from hypothesis.
ooooohhh, a BaseExceptionGroup
containing the StopTest
would indeed explain this.
In the long term I'd really like Hypothesis to "just work" with (Base)ExceptionGroups; in the short term we might end up just catching and crashing on this as a more-interpretable improvement on the status quo. We do already depend on the exceptiongroup
backport, but missing the except*
syntax before py311 is kinda painful...
from hypothesis.
Related Issues (20)
- Handle Django upgrades like Python versions in `./build.sh upgrade-requirements`
- Failing test for Django 5.0 HOT 1
- Using `builds` arguments for reprs may produce worse results than pretty printing HOT 2
- Improve error message when a package only has submodules for ghostwriter HOT 3
- Improve support for new and user-defined Numpy dtypes (e.g. `np.dtypes.StringDType`)
- Change Flaky to be an ExceptionGroup
- Error while patching time.perf_counter HOT 4
- Thread safety when usage with Pytest, and Coverage cause unexpected hang at exit HOT 3
- Falsifying examples not being printed HOT 8
- System.AccessViolationException with Hypothesis 6.108.x HOT 3
- Improve pretty-printer for overly long integers and signalling NaN
- Support `fuzz_one_input` on instance methods
- test.ghostwriter.py:test_obj_name fails because of tmp dir HOT 3
- Investigate `Examples` performance
- Python 3.14: StructuralCoverageTag.__init__() takes 1 positional argument but 2 were given HOT 9
- Improve support of `TypedDict`
- Support `LiteralString`
- Question: Write strategies to fuzz docstrings matching different styles (google, numpydoc, sphinx) HOT 3
- Support `TypeIs`
- Support `TypeVar` default
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hypothesis.