RuleBasedStateMachine tests do not work and are not documented about hypofuzz

[Zac-HD]

If someone is going to spend a few hours on this, we should probably add the logic necessary to support RuleBasedStateMachine tests instead of working around that - it's a bit fiddly but not that hard, and we'll definitely want it before HypoFuzz graduates from alpha to beta 🙂

[pspacek]

Additional note, after commenting out print around catch-all except I get more detailed traceback:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 273, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/logging.py", line 783, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/warnings.py", line 118, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1367, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 337, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 810, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/interface.py", line 58, in pytest_collection_finish
INTERNALERROR>     fuzz = FuzzProcess.from_hypothesis_test(
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/hy.py", line 96, in from_hypothesis_test
INTERNALERROR>     given_kwargs=wrapped_test.hypothesis._given_kwargs,
INTERNALERROR>                  ^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'function' object has no attribute 'hypothesis'

[eivindjahren]

It was also confusing to me why my tests were being skipped, and after debugging the HypoFuzz _ItemCollector I realized it was because the function was silently using the monkeypatch fixture through an autouse fixture. Since the monkeypatch fixture is not autouse, but automatically applied by the autouse fixture, all tests were skipped.