from_type fails on nested annotated type with mysterious error about hypothesis

[td-anne]

Aha! It seems things like IsFinite are not meant to be used in annotations at all, they are meant to be (parametrized) types on their own: annotated-types/annotated-types#61 (comment)

So maybe hypothesis is doing everything right. It would be nice to have tests of these in the test suite, though.

[td-anne]

As an additional data point, the following works as expected:

FiniteFloat = Annotated[float, Predicate(math.isfinite)]


@given(thing=st.from_type(FiniteFloat))
def test_annotated_type(thing):
    assert thing == thing

This is peculiar as annotated-types claims that IsFinite is defined to be Predicate(math.isfinite).

[JonathanPlasse]

Annotated[float, IsFinite] is not a type, if st.from_type accepts annotation like this its type signature should be updated.
Here is a similar issue raised on Pyright microsoft/pyright#7238.

from typing import Annotated

assert isinstance(float, type)
assert not isinstance(Annotated[float, "some annotation"], type)

[td-anne]

hypothesis/hypothesis-python/tests/test_annotated_types.py

Line 78 in 21fd917

def test_annotated_type_int(annotated_type, expected_strategy_repr):

This test suggests that this is the intended use of from_type; perhaps the name is simply inaccurate?

[td-anne]

In fact, it looks like the problem is that annotated-types tries to express a constraint on the kinds of type that can be annotated with IsFinite and hypothesis can't cope with this:

_NumericType = TypeVar('_NumericType', bound=Union[SupportsFloat, SupportsIndex])
IsFinite = Annotated[_NumericType, Predicate(math.isfinite)]

annotated-types also does this for its string predicates (e.g. IsLower), but hypothesis contains no tests of these either. And indeed these fail too:

AsciiString = Annotated[str, IsAscii]


@given(thing=st.from_type(AsciiString))
def test_annotated_type(thing):
    assert thing == thing

[td-anne]

It looks like the intended usage is as IsAscii[str] (https://github.com/annotated-types/annotated-types/blob/main/annotated_types/test_cases.py#L125) but this fails with the same error:

AsciiString = Annotated[str, IsAscii[str]]


@given(thing=st.from_type(AsciiString))
def test_annotated_type(thing):
    assert thing == thing

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

annotated_type = typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]]

    def find_annotated_strategy(annotated_type):
        metadata = getattr(annotated_type, "__metadata__", ())
    
        if any(is_annotated_type(arg) for arg in metadata):
            # We are in the case where one of the metadata argument
            # is itself an annotated type. Although supported at runtime,
            # This shouldn't be allowed: we prefer to raise here
>           raise ResolutionFailed(
                f"Failed to resolve strategy for the following Annotated type: {annotated_type}."
                "Arguments to the Annotated type cannot be Annotated."
            )
E           hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]].Arguments to the Annotated type cannot be Annotated.

../../Library/Caches/pypoetry/virtualenvs/da-dropper-wire-data-l5CQQYIb-py3.11/lib/python3.11/site-packages/hypothesis/strategies/_internal/types.py:310: ResolutionFailed

[Zac-HD]

I think the core problem here is that the error message is confusing, and we should improve it! e.g. you saw

• hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[float, typing.Annotated[~_NumericType, Predicate(func=<built-in function isfinite>)]].Arguments to the Annotated type cannot be Annotated.
• hypothesis.errors.ResolutionFailed: Failed to resolve strategy for the following Annotated type: typing.Annotated[str, typing.Annotated[str, Predicate(func=<method 'isascii' of 'str' objects>)]].Arguments to the Annotated type cannot be Annotated.

but "Arguments to the Annotated type cannot be Annotated" only helps if you already know the problem. Some ideas for how to improve this, in both Hypothesis and annotated-types:

• Fix the incorrect annotated-types README; IsFinite and other types are mis-described 😱 annotated-types/annotated-types#61 (comment)
• Improve the repr of Predicate (func=<method 'isascii' of 'str' objects> is... suboptimal)
• Improve the error message Hypothesis gives in this case.
  • General template: {the type} is invalid because {explanation}
  • Replace typing.Annotated with Annotated for brevity; it's obviously implied
  • Explain what a valid type would look like, e.g. nesting Annotated is allowed for the first (type) argument, but not for later (metadata) arguments.. Try to show what a valid type would look like in a "did you mean..." message, accounting for possibly identical/subclass/typevar inner types. Check in the annotated_types namespace and use the declared name if it's one of those objects. e.g.:
    • Annotated[float, IsFinite] is invalid because nesting Annotated is allowed for the first (type) argument, but not for later (metadata) arguments. Did you mean: Annotated[IsFinite[Float]? (OK, so we'll want to unwrap a no-other-metadata Annotated and actually just say IsFinite[float])
    • (imagining that you defined your own IsAscii type) Annotated[str, Annotated[str, Predicate(func=str.isascii)], something_else] is invalid because <snip>. Did you mean: Annotated[str, Predicate(func=str.isascii), something_else]?