GithubHelp home page GithubHelp logo

antonagestam / phantom-types Goto Github PK

View Code? Open in Web Editor NEW
192.0 5.0 9.0 1.92 MB

Phantom types for Python.

Home Page: https://pypi.org/project/phantom-types/

License: BSD 3-Clause "New" or "Revised" License

Python 99.35% Makefile 0.65%
phantom-types static-analysis mypy python static-typing typing refined-types refinement-types refined python3

phantom-types's Introduction

phantom-types

CI Build Status Documentation Build Status Test coverage report
PyPI Package Python versions

Phantom types for Python will help you make illegal states unrepresentable and avoid shotgun parsing by enabling you to practice "Parse, don't validate".

Installation

$  python3 -m pip install phantom-types

Extras

There are a few extras available that can be used to either enable a feature or install a compatible version of a third-party library.

Extra name Feature
[dateutil] Installs python-dateutil. Required for parsing strings with TZAware and TZNaive.
[phonenumbers] Installs phonenumbers. Required to use phantom.ext.phonenumbers.
[pydantic] Installs pydantic.
[hypothesis] Installs hypothesis.
[all] Installs all of the above.
$  python3 -m pip install phantom-types[all]

Examples

By introducing a phantom type we can define a pre-condition for a function argument.

from phantom import Phantom
from phantom.predicates.collection import contained


class Name(str, Phantom, predicate=contained({"Jane", "Joe"})):
    ...


def greet(name: Name):
    print(f"Hello {name}!")

Now this will be a valid call.

greet(Name.parse("Jane"))

... and so will this.

joe = "Joe"
assert isinstance(joe, Name)
greet(joe)

But this will yield a static type checking error.

greet("bird")

To be clear, the reason the first example passes is not because the type checker somehow magically knows about our predicate, but because we provided the type checker with proof through the assert. All the type checker cares about is that runtime cannot continue executing past the assertion, unless the variable is a Name. If we move the calls around like in the example below, the type checker would give an error for the greet() call.

joe = "Joe"
greet(joe)
assert isinstance(joe, Name)

Runtime type checking

By combining phantom types with a runtime type-checker like beartype or typeguard, we can achieve the same level of security as you'd gain from using contracts.

import datetime
from beartype import beartype
from phantom.datetime import TZAware


@beartype
def soon(dt: TZAware) -> TZAware:
    return dt + datetime.timedelta(seconds=10)

The soon function will now validate that both its argument and return value is timezone aware, e.g. pre- and post conditions.

Pydantic support

Phantom types are ready to use with pydantic and have integrated support out-of-the-box. Subclasses of Phantom work with both pydantic's validation and its schema generation.

class Name(str, Phantom, predicate=contained({"Jane", "Joe"})):
    @classmethod
    def __schema__(cls) -> Schema:
        return super().__schema__() | {
            "description": "Either Jane or Joe",
            "format": "custom-name",
        }


class Person(BaseModel):
    name: Name
    created: TZAware


print(json.dumps(Person.schema(), indent=2))

The code above outputs the following JSONSchema.

{
  "title": "Person",
  "type": "object",
  "properties": {
    "name": {
      "title": "Name",
      "description": "Either Jane or Joe",
      "format": "custom-name",
      "type": "string"
    },
    "created": {
      "title": "TZAware",
      "description": "A date-time with timezone data.",
      "type": "string",
      "format": "date-time"
    }
  },
  "required": ["name", "created"]
}

Development

Install development requirements, preferably in a virtualenv:

$ python3 -m pip install .[all,test]

Run tests:

$ pytest
# or
$ make test

Linting and static type checking is setup with pre-commit, after installing it you can setup hooks with the following command, so that checks run before you push changes.

# configure hooks to run when pushing
$ pre-commit install -t pre-push
# or when committing
$ pre-commit install -t pre-commit
# run all checks
$ pre-commit run --all-files
# or just a single hook
$ pre-commit run mypy --all-files

In addition to static type checking, the project is setup with pytest-mypy-plugins to test that exposed mypy types work as expected, these checks will run together with the rest of the test suite, but you can single them out with the following command.

$ make test-typing

phantom-types's People

Contributors

antonagestam avatar flaeppe avatar g-as avatar llyaudet avatar ramnes avatar sobolevn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

phantom-types's Issues

Phantom types for mutable base types

As discussed in #127 (comment) I am openning a new issue.

This code is very risky one to write:

from django.db import models

class  User(models.Model):
    is_paid = models.BooleanField(default=False)

def is_paid_user(value: object) -> bool:
    return isinstance(value, User) and value.is_paid

class PaidUser(User, Phantom, predicate=is_paid_user):
    ...  # metaclass conflict

Why? Because User is mutable. At anytime you can just user.is_paid = True | False to change its PaidUser instance check without mypy noticing the change.

We need to document this problem.

Explode partials in reprs

#148 introduced nice reprs for the shipped predicates and predicate factories. We can go a bit further and also implement specialized support for functools.partial objects.

PoC

>>> po = partial(str.split, sep=".", maxsplit=3)
>>> f"{po.func.__qualname__}({', '.join(po.args)}, {', '.join(f'{k}={v!r}' for k, v in po.keywords.items())})"
"str.split(, sep='.', maxsplit=3)"

Excellent pydantic support for sized containers

Currently there are two issues:

  • Pydantic doesn't expose the "sub-field" type to __modify_schema__ so we can't add e.g. {items: {"type": "integer"}} for a NonEmpty[int]. See discussion. Fixed in pydantic/pydantic#3434.
  • The PhantomSized type is "too" flexible. Consider adding an intermediary BoundedSize or similar that takes class arguments min/max and reflects that in minItems/maxItems.

Consider refactoring the `.parse()` method into a typeclass

https://github.com/dry-python/classes

  • Figure out if a typeclasses are compatible with phantom types
  • Come up with a reasonable API that is optimally backwards compatible the .parse() method. Is there value in allowing them to co-exist?

The background is I want to introduce dateutil-backed parsing for TZAware, but I don't want to make dateutil a required dependency. So the idea is to only define the parse function for TZAware when dateutil is installed. The type could still be usable without its parsing capabilities, e.g. to exclude non-tz-aware datetimes without having the ability to parse string-formatted timestamps.


Further reasoning: perhaps the .parse() method of every phantom type should be a typeclass? That way, by default TZAware.parse() would be implemented for datetime.datetime. When dateutil is installed TZAware.parse() would also be implemented for str and int.

AttributeError: 'operator.attrgetter' object has no attribute '__qualname__'

>>> compose2(print, attrgetter("a.b"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/anton/.pyenv/versions/gina-api/lib/python3.9/site-packages/phantom/fn.py", line 26, in compose2
    b_name = _name(b)
  File "/Users/anton/.pyenv/versions/gina-api/lib/python3.9/site-packages/phantom/fn.py", line 9, in _name
    return fn.__qualname__
AttributeError: 'operator.attrgetter' object has no attribute '__qualname__'

This should fall back to __name__ and when that doesn't exist either, should fall back to __str__().

Idea: something like `PreservedIterator`

Sometimes I have several layers for Iterator calls, which look like this:

def first() -> Iterator[int]:
    yield from range(5)

def second() -> Iterator[int]:
    yield from first()

def third() -> Iterator[int]:
    yield from second()

Sometimes, people can break things unintentionally. For example, imagine that second has this line now:

def second() -> Iterator[int]:
    gen = first()
    print(next(gen))
    yield from gen

Now, second will break the chain and produce one value less than first. Sometimes we want to explicitly disallow that.
Is there a way to express this using phantom-types? If no, let's consider adding a support for it:

def second() -> PreservedIterator[int]:
    gen = first()
    print(next(gen))  # error
    yield from gen

Usage with other types with metaclasses / metaclass conflicts

While working on #123 and solving a metaclass conflict, I was wondering about potential usage of phantom together with django / sqlalchemy projects that make a heavy use of metaclasses.

For example, this code will produce a metaclass conflict:

from django.db import models

class  User(models.Model):
    is_paid = models.BooleanField(default=False)

def is_paid_user(value: object) -> bool:
    return isinstance(value, User) and value.is_paid

class PaidUser(User, Phantom, predicate=is_paid_user):
    ...  # metaclass conflict

Is there any simple way to support this?
We can probably ship some util metaclass solver like the one from here: https://stackoverflow.com/a/41266737/4842742

Or at least we can document how to solve this.

Parse mutable sized containers into tuple/immutables.Map

Something along these lines, but recursive:

    @classmethod
    def parse(cls: Derived, instance: object) -> Derived:
        if isinstance(instance, mutable):
            return super().parse(immutables.Map(instance) if isinstance(instance, Mapping) else tuple(instance))
        return super().parse(instance)

It might seem like using Hashable to deem objects immutable would be a good idea, see for instance https://docs.python.org/3.9/glossary.html#term-hashable

... but it's not:

class A: ...
hash(A())  # this is not an error ...

Trying to set up local development, but can't run YAML tests

Hi! I heard about this library from typing-sig and it looks really interesting. I tried setting up local development and ran into a couple issues.

  1. When I run pre-commit run --all, there's a flake8 error:
(env) ➜  phantom-types git:(main) ✗ pre-commit run --all
Check for case conflicts.................................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
Debug Statements (Python)................................................Passed
Detect Private Key.......................................................Passed
pyupgrade................................................................Passed
autoflake................................................................Passed
isort....................................................................Passed
black....................................................................Passed
blacken-docs.............................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

src/phantom/fn.py:36:9: B018 Found useless expression. Either assign it to a variable or remove it.

Validate GitHub Workflows................................................Passed
mypy.....................................................................Passed
check-manifest...........................................................Passed
format-readme............................................................Failed
- hook id: format-readme
- exit code: 1

Executable `docker` not found

(env) ➜  phantom-types git:(main) ✗

And indeed, looking at line 36 of that file shows an unassigned string expression. I assume this is just leftover since that line hasn't been touched since April, but maybe I'm misunderstanding.

  1. Running make test and/or pytest fails on all of the yaml-based tests. Here's an example:
(env) ➜  phantom-types git:(main) ✗ pytest tests/ext/test_phonenumbers.yaml
============================================================================== test session starts ==============================================================================
platform darwin -- Python 3.9.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /Users/tushar/code/phantom-types, configfile: pyproject.toml
plugins: mypy-plugins-1.9.2, typeguard-2.13.3
collected 3 items

tests/ext/test_phonenumbers.yaml FFF                                                                                                                                      [100%]

=================================================================================== FAILURES ====================================================================================
_____________________________________________________________________________ bound_is_not_subtype ______________________________________________________________________________
/Users/tushar/code/phantom-types/tests/ext/test_phonenumbers.yaml:9:
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output:
E   Actual:
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:53: error: Unused "type: ignore[misc]" comment (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:144: error: ClassVar cannot contain type variables  [misc] (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:162: error: Unused "type: ignore" comment (diff)
E     main:7: error: Argument 1 to "takes_phone_number" has incompatible type "str"; expected "PhoneNumber"  [arg-type] (diff)
E     main:12: error: Argument 1 to "takes_formatted_phone_number" has incompatible type "str"; expected "FormattedPhoneNumber"  [arg-type] (diff)
E   Expected:
E     main:7: error: Argument 1 to "takes_phone_number" has incompatible type "str"; expected "PhoneNumber"  [arg-type] (diff)
E     main:12: error: Argument 1 to "takes_formatted_phone_number" has incompatible type "str"; expected "FormattedPhoneNumber"  [arg-type] (diff)
E   Alignment of first line difference:
E     E: main:7: error: Argument 1 to "takes_phone_number" has incompatible type ...
E     A: ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:53: e...
E        ^
________________________________________________________________________________ can_instantiate ________________________________________________________________________________
/Users/tushar/code/phantom-types/tests/ext/test_phonenumbers.yaml:16:
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Output is not expected:
E   Actual:
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:53: error: Unused "type: ignore[misc]" comment (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:144: error: ClassVar cannot contain type variables  [misc] (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:162: error: Unused "type: ignore" comment (diff)
E   Expected:
E     (empty)
___________________________________________________________________________________ can_infer ___________________________________________________________________________________
/Users/tushar/code/phantom-types/tests/ext/test_phonenumbers.yaml:30:
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Output is not expected:
E   Actual:
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:53: error: Unused "type: ignore[misc]" comment (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:144: error: ClassVar cannot contain type variables  [misc] (diff)
E     ../../../../../../Users/tushar/code/phantom-types/src/phantom/base:162: error: Unused "type: ignore" comment (diff)
E   Expected:
E     (empty)
============================================================================ short test summary info ============================================================================
FAILED tests/ext/test_phonenumbers.yaml::bound_is_not_subtype -
FAILED tests/ext/test_phonenumbers.yaml::can_instantiate -
FAILED tests/ext/test_phonenumbers.yaml::can_infer -
=============================================================================== 3 failed in 0.81s ===============================================================================
(env) ➜  phantom-types git:(main) ✗

It looks to me like every test is throwing errors on lines 53, 144, and 162 of src/phantom/base.py, e.g., here. If I run mypy ., the same errors appear in the output there.

The first test example (bound_is_not_subtype) shows that I am getting the expected results, but every test has these three extra errors appended to it. I assume this is a difference in mypy config that I'm missing, but I'd love help figuring this out.

Thanks for your great work on this library!

Check for mutable bounds in Phantom or PhantomBase

Related to #128, I'm realizing there's more we can do to eliminate the risk of users using phantom types with mutable bounds. The check from sized can be reused but made to raise at import-time. It would also make sense to check for non-frozen dataclasses, and perhaps even to check dataclass fields.

TypedDict support

Right now if you want to write a TypedDict that will work with isinstance - you need to write:

from typing_extensions import TypedDict

class User(TypedDict):
    name: str
    registered: bool

class UserDictMeta(type):
    def __instancecheck__(cls, arg: object) -> bool:
        return (
            isinstance(arg, dict) and
            # The part below can be really long if `user` has lots of fields
            isinstance(arg.get('name'), str) and
            isinstance(arg.get('registered'), bool)
        )

UserMeta = type('UserMeta', (UserDictMeta, type(TypedDict)), {})

class UserDict(User, metaclass=UserMeta):
    ...

a = {'name': 'sobolevn', 'registered': True}
print(isinstance(a, UserDict))  # True

I see that people can benefit from a better API, like:

from phantom.structure import TypedDictWithStructure
from typing_extensions import TypedDict

class User(TypedDict):
    name: str
    registered: bool

class UserDict(User, TypedDictWithStructure):
   ...

a = {'name': 'sobolevn', 'registered': True}
print(isinstance(a, UserDict))  # True

Possible problems:

  1. #128
  2. mypy right now does not allow to call isinstance with TypedDict subtypes: error: Cannot use isinstance() with TypedDict type

What do you think?

Ship a documented utils.compose?

To turn a pretty simple check like this:

def is_valid_event_name(name: str) -> bool:
    parts = name.split(".")
    return len(parts) == 3 and all(part.isidentifier() for part in parts)

... into a composed function, one pretty quickly bumps into the lack of functional composition in Python. Should phantom-types ship one?

Address lacking coverage

From latest PR:

Name                  Stmts   Miss Branch BrPart  Cover   Missing
-----------------------------------------------------------------
phantom/base.py          80      2     26      2    96%   26->27, 27, 98->105, 105
phantom/interval.py      26      1      2      1    93%   34->35, 35
phantom/re.py            10      1      0      0    90%   15
-----------------------------------------------------------------
TOTAL                   374      4     42      3    98%

These are probably all cases where a 100% test coverage makes sense.

Consider

Missing a "Resources" or "More examples" section/page

Hello! I'm willing to try this library with some dummy code I have with Pydantic and other serialization/validation libraries. However, I cannot get pass the example of using str. For example, I don't get why the following happens:

from typing import List
from python.sized import NonEmpty

it_works = NonEmpty[str].parse("hello")
it_doesnt_work = NonEmpty[List[int]].parse([1,2])

Is there any additional example with container types, or a recorded talk/workshop showing how this works? Thanks in advance!

Implement collections.every

def every(predicate: Predicate[object]) -> Predicate[Iterable]:
    """
    Return a predicate that is successful given an iterable where all items satisfy `predicate`.
    """

    def compare(iterable: Iterable) -> bool:
        return all(predicate(item) for item in iterable)

    return compare

`NonEmpty[str]` type cannot be used as sorting key

Not sure if I'm using NonEmpty incorrectly here with str. But the code below does not typecheck, and I think it should? (Since the underlying data type is a string)

from typing import NamedTuple
from phantom.sized import NonEmpty


class X(NamedTuple):
    value: NonEmpty[str]

sorted(
    [X(value=NonEmpty[str].parse("first")), X(value=NonEmpty[str].parse("second"))],
    key=lambda x: x.value
)

Results in the following error

10: error: Argument "key" to "sorted" has incompatible type "Callable[[X], NonEmpty[str]]"; expected "Callable[[X], Union[SupportsDunderLT, SupportsDunderGT]]"  [arg-type]

10: error: Incompatible return value type (got "NonEmpty[str]", expected "Union[SupportsDunderLT, SupportsDunderGT]")  [return-value]

Schema for `iso3166.CountryCode` is not OpenAPI 3.0 compatible (?)

I have a pydantic model set up like this:

class MyModel(pydantic.BaseModel):
    country_code: CountryCode

Then I run a OpenAPI schema validator against the generated schema found here: https://github.com/p1c2u/openapi-spec-validator

Getting an issue in the anyOf below, on examples. Not sure if it's misplaced?

{
  "type": "string",
  "title": "Alpha2",
  "description": "ISO3166-1 alpha-2 country code",
  "examples": [
    "NR",
    "KZ",
    "ET",
    "VC",
    "AE",
    "NZ",
    "SX",
    "XK",
    "AX"
  ],
  "format": "iso3166-1 alpha-2"
}

Anyways, removing examples from there everything works fine. Perhaps it should move someplace else? I tried moving it on the "root" level of the field object (sibling to anyOf) but that didn't work either.

Make `TZAware.parse` use dateutil

If dateutil is installed we should use it to parse strings in TZAware.parse.

It will also make sense to make it use datetime.datetime.fromtimestamp for ints.

It might be necessary to take infinity into consideration, see this issue.

Add tests for bound erasure for shipped types

Make sure access to methods of the bound type doesn't raise mypy errors.

This fix solves that for the TZ types, there should be mypy tests asserting that cast(TZAware, object()).isoformat() doesn't raise a attr-doesn't-exist error.

Reintroduce support for Python 3.7

Hi! A quick question:

Why have you dropped python3.7 support in this commit c0817ed ?

Looks like python3.7 was supported not a long time ago: c2e6d37

From what I can see the only feature of python3.8 you are using is / arguments, which can be easily converted into regular arguments (which is even cleaner in my opinion). Anything else that I've missed?

Support intersections of concrete types

class A(Sequence, Phantom, predicate=contains("a")):
    ...
class B(Iterable, Phantom, predicate=contains("b")):
    ...
class C(A, B):
    ...
  • Bound of C should be (Sequence, Iterable) (intersection).
  • Predicate of C should be bool.both(contains("a"), contains("b"))

Improve documentation

  • Rename "Getting started" to "Background" /"Why use phantom types?"
  • Write a new Getting started section focusing on using predicate functions.
  • Add section about bounds
  • Write a small example to include in readme.
  • Address todos.
  • Move predicate and type descriptions into docstrings and use autodoc.
  • Use a theme that renders autodocs nicely.
  • Build docs in CI.
  • Document Phantom.
  • (Fix headers in README ...)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.