ltworf / typedload Goto Github PK
View Code? Open in Web Editor NEWPython library to load dynamically typed data into statically typed data structures
Home Page: https://ltworf.codeberg.page/typedload/
License: Other
Python library to load dynamically typed data into statically typed data structures
Home Page: https://ltworf.codeberg.page/typedload/
License: Other
In [21]: load(None, Optional[str])
...:
Out[21]: 'None'
This happens.
Should NoneType
be a basic type?
They gave me a limited amount of minutes to run tests, and after that they will want to be paid, so evaluate and migrate to something else is necessary.
If a field has a list factory and the value is empty, the field is dumped with hidedefaults=True.
Version 3 exists solely for the deferred type evaluation feature of python 3.10.
However that change is for now being reverted, and pushed on to an unknown time in the future.
So it makes no sense to break API at this point. Revert the changes and make a 2.8 version instead.
List[Tuple[Callable[[Any], bool], Callable[[Loader, Any, Type], Any]]]
You wrote in your README
Note that it is released with a GPL license and it cannot be used inside non GPL software.
What is your reason that your code under GPL can't be used in non-GPL software?
Could it be possible to mention apischema as an alternative in the comparison section of the documentation?
Had to go back 1 version
json = Union[int, float, str, bool, None, List['json'], Dict[str, 'json'] ]
data = [{},[]]
assert typedload.load(data, json, frefs={'json': json}, basiccast= False) == data
This code, it fails or succeeds depending on the order of the arguments of the union, which changes between different executions of python.
It might be better to extend the initial check about basic types to avoid doing a cast, if that is possible.
This would mean that when loading something into Union[List[t], Tuple[t]]
, if the data is a list, it would always pick the list, and if it's a tuple, it'd pick the tuple.
Literal types. See PEP 586 and typing.Literal. Literal types indicate that a parameter or return value is constrained to one or more specific literal values:
def get_status(port: int) -> Literal['connected', 'disconnected']:
...
This seems kinda the same as an enum but without the names for the values.
I have some things that are a Dict[str, Any]
where any can be any thing representable in json. So I basically want typeshed to "pass through" the values of that dict. I'm now using Dict[str, Union[str, int, float, bool, NoneType]]
but to make it work recursively the type is even longer.
TypeError: Cannot deal with value {'x': 5, 'y': 6, 'next': None} of type _ForwardRef('Node')
class Node(NamedTuple):
x: int = 1
y: int = 2
next: 'Node' = None
They can be useful when writing code that needs to check types and run on multiple python versions.
The annotations are useful to track specifically which element is causing the issue. All the functions calling l.load()
should have annotations.
Hey, I have a behaviour which I think is a bug / missing functionality.
Here is the code (using typedload==2.7
):
from typing import Optional, TypedDict
import typedload
class ExampleTypedDict(TypedDict):
a: int
b: Optional[int]
d = {"a": 0}
print(f"{d=}")
td = ExampleTypedDict(**d)
print(f"{td=}")
tdl = typedload.load(d, ExampleTypedDict)
And here is the result of running it:
➜ optional_typeddict pipenv run python main.py
d={'a': 0}
td={'a': 0}
Traceback (most recent call last):
File "/Users/kkom/Desktop/optional_typeddict/main.py", line 17, in <module>
tdl = typedload.load(d, ExampleTypedDict)
File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/__init__.py", line 200, in load
return loader.load(value, type_)
File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 294, in load
raise e
File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 290, in load
return cast(T, func(self, value, type_))
File "/Users/kkom/.local/share/virtualenvs/optional_typeddict-hmzsbyx_/lib/python3.9/site-packages/typedload/dataloader.py", line 500, in _namedtupleload
raise TypedloadValueError(
typedload.exceptions.TypedloadValueError: Value does not contain fields: {'b'} which are necessary for type ExampleTypedDict
Path: .
PS: Here is my exact setup:
➜ optional_typeddict cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
typedload = "==2.7"
[dev-packages]
[requires]
python_version = "3.9"
CC: @sid-at-github
If a field is defined as a compound type, e.g. List[T]
or Dict[str,T]
, but the JSON contents specifies null
as the value for the entire field, typedload.load(...)
should detect that explicitly and generate an error along the lines of "null can not be converted to ..."
. The current behavior generates confusing errors (see below, I only checked lists and dictionaries).
List[T]
The actual error thrown:
AssertionError()
What really happens is that enumerate(...)
can not be called on a None
object here:
Traceback (most recent call last):
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 216, in load
return func(self, value, type_)
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 273, in _listload
return [l.load(v, t, annotation=Annotation(AnnotationType.INDEX, i)) for i, v in enumerate(value)]
TypeError: 'NoneType' object is not iterable
Dict[str,T]
The actual error thrown:
TypedloadAttributeError("'NoneType' object has no attribute 'items'")
What really happens is that .items()
can not be called on a None
object here:
Traceback (most recent call last):
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 286, in _dictload
for k, v in value.items()}
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 286, in <dictcomp>
for k, v in value.items()}
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 220, in load
raise e
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 216, in load
return func(self, value, type_)
File "c:\program files\python37\lib\site-packages\typedload\dataloader.py", line 288, in _dictload
raise TypedloadAttributeError(str(e), type_=type_, value=value)
typedload.exceptions.TypedloadAttributeError: 'NoneType' object has no attribute 'items'
Value: None
Type: typing.Dict[str, typing.List[str]]
Load trace:
Type: typing.Dict[str, typing.List[str]] Annotation: (AnnotationType.VALUE None) Value: None
Path:
Great library, thanks for creating and sharing it!
I wanted to report that I had problems when using it with dates and datetimes:
from datetime import date
from typing import NamedTuple
import typedload
class Point(NamedTuple):
x: float
y: float
created_at: date
print(
typedload.dump(
Point(x=5.0, y=6.0, created_at=date(2019, 1, 1))
)
)
It crashes with:
raise TypedloadValueError('Unable to dump %s' % value, value=value)
typedload.exceptions.TypedloadValueError: Unable to dump 2019-01-01
Value: 2019-01-01
Type: None
Not sure if this is a bug, a feature, or simply not implemented yet.
[12:34] <mapreri> LtWorf: perchè non build-dependi su python3-all?
[12:34] <mapreri> mh, comunque non fai i test in loop
[12:34] <LtWorf> i test in loop?
[12:35] <LtWorf> cioè?
[12:35] <mapreri> LtWorf: le cose python di solito vengono compilate e testate per tutte le versioni di python(3) supportate. questo pacchetto per come è fatto lo fa solo per la versione di default
[12:36] <LtWorf> hm, ho una continuous integration su github che fa sta cosa cmq
[12:36] <mapreri> beh, è ortogonale
[12:36] <LtWorf> cmq ok, se mi dici un pacchetto che fa così, vedo come funziona
[12:37] <mapreri> ma dato che quel i test non sono cosi "standard" con pytest potrebbe non essere così immediato
[12:37] <mapreri> beh, quasi tutti quelli che usano pybuild e non fanno override di dh_auto_test
[12:37] <mapreri> e build-dependono su python3-all
[12:38] <mapreri> https://salsa.debian.org/python-team/modules/python-psutil/blob/master/debian/rules questo è uno che non usa pybuild ma fa quella cosa
[12:38] <LtWorf> ma uso pybuild io
[12:38] <mapreri> vedi come sia tutto in un loop `for python in $(PY2VERS) $(PY3VERS);` etc?
[12:38] <mapreri> pybuild è soltanto glue code che fa quel loop
[12:38] <LtWorf> posso fare che nell'override chiama dh_test, e fa anche "make mypy"
[12:39] <mapreri> più che altro, in quei `make test` e `make mypy` puoi specificare che interprete usare?
[12:39] <mapreri> perchè ora c'è solo py3.6 quindi è anche difficile da testare per bene, la differenza la vedi quando arriverà py 3.7
[12:40] <LtWorf> lo so, già ho provato manualmente con 3.7, ed è un casino
[12:40] <LtWorf> anche perché quello che hanno su travis è una versione diversa da quello che c'è in debian, e già hanno API diverse
[12:40] <mapreri> joy :D
[12:40] <LtWorf> ste API dei tipi cambiano a cazzo, su versioni minori
[12:41] <LtWorf> cmq vabbè, vedo di fare sta cosa allora
[12:41] <mapreri> |.PHONY: test
[12:41] <mapreri> |test:
[12:41] <mapreri> | python3 -m tests
[12:41] <mapreri> (dal makefile) quindi direi che non puoi farlo
[12:41] <mapreri> nel senso, richiede un attimo di "muovere cose in giro"
[12:41] <LtWorf> si, usa solo il python3 di default
[12:41] <mapreri> carico questo, ci penserai per il futuro non definito?
[12:41] <LtWorf> ma potrei usare una variabile nel makefile per passare l'interprete
[12:41] <mapreri> sì, qualcosa del genere
When loading a huge amount of items into a list it would probably cause a great speed up. Maybe do some tests to see how much would that be.
Just presume it's an empty dictionary
It would be easy to support name mangling like for attr.
@ltworf - this particular code in _namedtupleload()
saved my butt today:
# Prepare the list of the needed name changes
transforms = [] # type: List[Tuple[str, str]]
for field in fields:
if type_.__dataclass_fields__[field].metadata:
name = type_.__dataclass_fields__[field].metadata.get('name')
if name:
transforms.append((field, name))
So, I truly take my hat to you for even thinking about this...
Having said that, it would be nice, if this particular feature were documented in the official guide. It took me a while to find this in the source code - I was actually trying to figure out how to implement this exact Python -> JSON mapping. Imagine my surprise, when I discovered that you have already thought about this :)
Once I found this code, it did not take me long to figure out that it is as simple as this:
@dataclass
class Foo():
foo_bar: str = field(metadata={"name": "foo-bar"})
But I am not sure if everybody will find that at all...
I still I owe you a beer :D
Having that will cause everything to fail. It looks like I want to support it.
On error, it'd be nice to print all the hierarchy, to know exactly where the error happened, in the data tree.
Hello
I'm a big fan of this library and have used it in personal projects, but can't use it at work.
I'd love to use at my company and contribute to this library, but we can't due to the copyleft. Is there a specific reason for GPL 3.0? Any chance a more permissive license could be used? MIT/BSD or LGPL?
try:
return cast(T, func(self, value, type_))
except Exception as e:
> assert isinstance(e, TypedloadException)
E AssertionError
../../../.local/lib/python3.8/site-packages/typedload/dataloader.py:236: AssertionError
I get this error in my application when calling typedload.load
and I'm not sure what I should do about it - seems like that assert is there to catch a bug in typedload?
So that NamedTuple when converted to dict
would have "no key" instead of None
.
Perhaps, Optional[]
will do?
has unrecognized fields and cannot be loaded into
This exception would be better if there was a list of which fields are too many.
The typing.NamedTuple class has deprecated the _field_types attribute in favor of the annotations attribute which has the same information. (Contributed by Raymond Hettinger in bpo-36320.)
Hi,
I am trying to load a sub-set of data into a data_class, but its not allowing me, is this not supported?
Are there any special flags I need to send in?
Error:
<class 'typedload.exceptions.TypedloadValueError'> Value does not contain fields: {'f1', 'f2', 'f3', 'f4'} which are necessary for type <class 'app.models.MyData'>
In your main example, you have sessions
with a default value of empty array. If you do not include sessions on your data, then it should assume empty array (the default) or at least have the option to.
Thanks
With the non pre evaluated types :(
document literal+union usage
Maybe link to mypy where it illustrates how that is done
For the parameters, so there is no need to set them later on.
Is it possible to change the license to one that does not requires the dependent codebase be open sourced?
.
Use this mypy configuration
[mypy]
python_version=3.5
warn_unused_ignores=True
warn_redundant_casts=True
strict_optional=True
scripts_are_modules=True
check_untyped_defs=True
https://mypy.readthedocs.io/en/latest/more_types.html#typeddict
It will be in the next python. Will just wait for it to be in typing instead.
Investigate and possibly fix.
A dictionary type with per-key types. See PEP 589 and typing.TypedDict. TypedDict uses only string keys. By default, every key is required to be present. Specify “total=False” to allow keys to be optional:
class Location(TypedDict, total=False):
lat_long: tuple
grid_square: str
xy_coordinate: tuple
List the value and the options (if they are few) in the exception.
field_defaults from the list.
Remove elif isinstance(value, tuple) and {'_fields', '_field_defaults', '_asdict'}.issubset(set(dir(value))):
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.