Improve discrimination of database keys about hypothesis

[Zac-HD]

I agree that we should pick up any low-hanging improvements here, and this list looks pretty good to me.

Specific to the last point, we have a magic attribute used to distinguish @pytest.mark.parametrize()d tests, which could also be used here. Just assign whatever bytestring you like, and it'll be hashed into the database key along with everything else, e.g.:

hypothesis/hypothesis-python/src/_hypothesis_pytestplugin.py

Lines 297 to 299 in 7b63483

	# Give every parametrized test invocation a unique database key
	key = item.nodeid.encode()
	item.obj.hypothesis.inner_test._hypothesis_internal_add_digest = key

[Zac-HD]

It's actually pretty hard to derive a stable-across-runs bytestring from an arbitrary strategy; for example it's easy to end up with a repr that includes some object's memory address, or less often something like the current time or date or machine name or OS.

We strip the decorators from source code before hashing it so that adding or removing @settings() and @example() decorators doesn't affect the database key.

[jobh]

Hmm, I think we might add to the list

• Pick up _hypothesis_internal_add_digest from the currently executing test function instead of the function actually passed to given.

otherwise we'd duplicate keys for f.x.

@pytest.mark.parametrize("v", [1, 2])
def test_something(v):

    @given(st.integers(v))
    def inner(x):
        assert x >= v

    inner()

since the attribute is added to test_something and not inner. At that point, it might be more straightforward to just store the add_digest as a global/threadlocal rather than as an attribute.

[edit] ...actually, we could mix in currently executing nodeid regardless of parametrization... and call it a day. Too pytest specific?

[Zac-HD]

[edit] ...actually, we could mix in currently executing nodeid regardless of parametrization... and call it a day. Too pytest specific?

Too specific - the problem is that if you execute a particular test function manually or via pytest, or a method with unittest or pytest, we want to have the same database key in either case. So mixing in the nodeid is fine when it's a parametrized test because there's a solid benefit and we're unlikely to execute it without pytest, but I'd prefer to avoid doing that unconditionally.

[jobh]

Understood, thanks! I have just one more question:

We don't use the strategy definition in our db key, and it looks intentional (e.g., _clean_source in get_digest and ignoring specifier in find()). This leads to collisions in find (naturally), and also f.x. test_attrs_inference_builds which is defined by the same name but slightly different strategy in two test modules.

Why is this? Is it beneficial to have key stability across changes in strategy?