dobiasd / undictify Goto Github PK
View Code? Open in Web Editor NEWPython library providing type-checked function calls at runtime
Home Page: https://pypi.org/project/undictify/
License: MIT License
Python library providing type-checked function calls at runtime
Home Page: https://pypi.org/project/undictify/
License: MIT License
This is related to #6, after having spent some time using the library in a bunch of production services.
"Default" type conversions are fine most of the times (with some exception - see #9), but there are several occasions in which you'd like to override the conversion function for a field.
This is usually dates, where the format has to be spelled out, but it could be as well classes with configuration parameters that you might want to capture into a lambda function using partial application.
As of today, what I usually do is writing a wrapper function that first takes care of the fields requiring custom conversion and then relies on undictify
to deal with the remaining fields.
Would you consider adding an option in type_checked_constructor
to specify a dictionary of field_name
-> conversion function
to override the default behaviour?
Here I come again ๐
I have been happily using undictify
and everything works properly... until I do:
from __future__ import annotations
in Python 3.7.
If I have a simple class like:
@type_checked_constructor(convert=True, skip=True):
class Example:
a: str
and I try to instantiate it I get a "str
is not a function" complaint, because I assume the annotations stops being callable. ๐ค
I believe the output should be: '3 * 2 == 6'.
The InitVar feature of dataclasses is not compatible with undictify.
InitVar's contained type should be checked as a special case for dataclasses. This will properly type-check the constructor.
See here for an explanation of InitVar.
from dataclasses import dataclass, InitVar, field
from undictify import type_checked_constructor
@type_checked_constructor()
@dataclass
class Hello:
x: int
y: int
my_init_var: InitVar[str]
z: str = field(init=False)
def __post_init__(self, my_init_var: str):
self.z = f"Hello, {my_init_var}"
print(Hello(x=11, y=2, my_init_var="hello"))
Result:
Traceback (most recent call last):
File "main.py", line 18, in <module>
print(Hello(x=11, y=2, my_init_var="hello"))
File "~/src/sandbox/fun-lsp/.venv/lib/python3.7/site-packages/undictify/_unpack.py", line 66, in wrapper
converters)
File "~/src/sandbox/fun-lsp/.venv/lib/python3.7/site-packages/undictify/_unpack.py", line 185, in _unpack_dict
converters)
File "~/src/sandbox/fun-lsp/.venv/lib/python3.7/site-packages/undictify/_unpack.py", line 246, in _get_value
raise TypeError(f'Key {param_name} has incorrect type: '
TypeError: Key my_init_var has incorrect type: str instead of InitVar.
I have been playing with the library today and one thing I didn't manage to get up and running were dates: I have a timestamp coming back in JSON response body and I want to get it into a NamedTuple field with type datetime
; even if convert=true
I can't manage to get it done - I feel I should be able to specify the function required to the conversion.
Do you have a way to handle this kind of cases?
@Dobiasd i really like the idea of your approach, especially the integration with dataclasses, and i works really well for simple cases as far as I tested.
However I'm in a situation, where I'd like to inherit from a base class decorated with type_checked_constructor and also have the same decorator on the derived class to. Right now in version 0.7.1 with python 3.6.8 it raises a TypeError('Class is already wrapped by undictify.')
Also would love to help with the implementation / testing, if that helps.
Just FYI, in case you'd like to adopt your forks, scripts, whatever. โ
I want to use undictify to deserialize JSON responses from APIs. I'm using Python 3.6.5, and I backported dataclasses from https://github.com/ericvsmith/dataclasses.
Check out this example:
from typing import Optional, Union, TypeVar
from dataclasses import dataclass
from undictify import type_checked_constructor
@type_checked_constructor(skip=True, convert=True)
@dataclass
class MyObject:
name: str
number: Optional[Union[int, float]]
works = {"name": "my_name", "number": 1.2}
my_object = MyObject(**works) # Works just fine
does_not_work = {"name": "my_name"}
my_object = MyObject(**does_not_work)
The desired behavior is that number
can be
int
float
None
But, when I try omitting number
, I get the following:
Traceback (most recent call last):
File "undictify_test.py", line 17, in <module>
my_object = MyObject(**does_not_work)
File "/usr/local/lib/python3.6/site-packages/undictify/_unpack.py", line 66, in wrapper
converters)
File "/usr/local/lib/python3.6/site-packages/undictify/_unpack.py", line 188, in _unpack_dict
return _unwrap_decorator_type(func)(first_arg, **call_arguments)
TypeError: __init__() missing 1 required positional argument: 'number'
I also tried the following:
Number = Union[int, float]
@type_checked_constructor(skip=True, convert=True)
@dataclass
class MyObject:
name: str
number: Optional[Number]
does_not_work = {"name": "my_name"}
my_object = MyObject(**does_not_work) # raises same exception as above
And
Number = TypeVar("Number", int, float)
@type_checked_constructor(skip=True, convert=True)
@dataclass
class MyObject:
name: str
number: Optional[Number]
works = {"name": "my_name", "number": 1.2}
my_object = MyObject(**works)
Using TypeVar raises:
File "undictify_test.py", line 16, in <module>
my_object = MyObject(**works)
File "/usr/local/lib/python3.6/site-packages/undictify/_unpack.py", line 66, in wrapper
converters)
File "/usr/local/lib/python3.6/site-packages/undictify/_unpack.py", line 185, in _unpack_dict
converters)
File "/usr/local/lib/python3.6/site-packages/undictify/_unpack.py", line 239, in _get_value
return func(value)
File "/usr/local/lib/python3.6/typing.py", line 194, in __call__
raise TypeError("Cannot instantiate %r" % type(self))
TypeError: Cannot instantiate typing.TypeVar
Any ideas? I'm a bit new to typing, so apologies if I'm doing something stupid.
I started using this library over the weekend to convert dictionaries (derived from JSON by the requests library) into dataclasses. It was working really well. However, when I pulled in your latest updates, unpack_dict
and unpack_json
were more or less gone. It appears as though the entire purpose of the package has shifted into type-checking, instead of unpacking dictionaries and dictionary-like strings (JSON's). Am I missing something here?
@Dobiasd, first of all: great job!
This is something I'd love to use at work and I'd like to contribute to make it production-ready - what is the current status of the library according to your judgement? Where do you need help or contributions?
Hello and thanks for this great library! We โค๏ธ it!
Currently I have this use case: I have a JSON like this:
{
"input": {
"events": [
{
"scheduleId": "U2NoZWR1bGU6MzYwYWY2NTItNmQwOC00YWZmLWEwNDYtOWNmOGQzYTE5ZDE4",
"type": "...",
"detail": "..."
}
]
}
}
I'm using the following dataclass to deserialize each event object:
@type_checked_constructor(
converters={
"schedule_id": _converter_global_id,
"type": _converter_enum(EventType),
}
)
@dataclass
class Event:
schedule_id: str
type: EnumType
detail: str
My problem is the following: each Event object has a schedule_id
which is given as BASE-64 string, my converter _converter_global_id
decodes the Base-64 and returns it but since the input is str
and output is also str
the "conversion" is skipped. I believe that decision is taken here: (derived from this conversation #10 (comment))
undictify/undictify/_unpack.py
Line 222 in 6dbed39
I was wondering if that's something we can override? I tried using the convert=True
flag with the same outcome ๐ข
Not quite sure why this is happening, but when using an Optional dataclass variable, the unpacker is blowing up. The deepest I managed to dig was that the traceback was referring the the Union object, but that it's only arguments were *args, **kwds
(see below for more reference).
import json
from typing import Callable, TypeVar, Optional
from dataclasses import dataclass
from undictify import type_checked_apply_skip_convert
TypeT = TypeVar('TypeT')
def unpack_json(target_func: Callable[..., TypeT], data: str) -> TypeT:
return type_checked_apply_skip_convert(target_func, **json.loads(data))
@dataclass
class Point:
x: int
y: int
@dataclass
class Rect:
left: Point
right: Point
top: Point
bottom: Optional[Point]
data_json = """
{
"label" : "square",
"left": {
"x": 3,
"y": 2,
"z": 1
},
"right": {
"x": 3,
"y": 2
},
"top": {
"x": 3,
"y": 2
},
"bottom": {
"x": 3,
"y": 2
}
}
"""
r = unpack_json(Rect, data_json)
print(r)
Traceback (most recent call last):
File "scratch.py", line 49, in <module>
r = unpack_json(Rect, data_json)
File "scratch.py", line 11, in unpack_json
return type_checked_apply_skip_convert(target_func, **json.loads(data))
File "/Users/shawalli/scratch/venv/lib/python3.7/site-packages/undictify/_unpack.py", line 90, in type_checked_apply_skip_convert
True, True)
File "/Users/shawalli/scratch/venv/lib/python3.7/site-packages/undictify/_unpack.py", line 153, in __unpack_dict
convert_types)
File "/Users/shawalli/scratch/venv/lib/python3.7/site-packages/undictify/_unpack.py", line 173, in __get_value
return __unpack_dict(target_type, value, convert_types)
File "/Users/shawalli/scratch/venv/lib/python3.7/site-packages/undictify/_unpack.py", line 136, in __unpack_dict
raise TypeError('Only parameters of kind POSITIONAL_OR_KEYWORD '
TypeError: Only parameters of kind POSITIONAL_OR_KEYWORD supported in target functions.
>>> func
typing.Union[__main__.Point, NoneType]
>>> param.kind
<_ParameterKind.VAR_POSITIONAL: 2>
>>> signature.parameters
mappingproxy(OrderedDict([('args', <Parameter "*args">), ('kwds', <Parameter "**kwds">)]))
I am trying to type check the initialisation of a class that I have decorated with @dataclass
:
@dataclass
@type_checked_constructor(skip=True, convert=False)
class Input:
transaction_id: str
but when I try to instantiate it with
payload = {
"transaction_id": "asdioansuiodb123asd"
}
Input(**payload)
I get an error back which I sense to be related to some compatibility issue between dataclass
and the undictify
wrapper. Can you help me to get to the bottom of it?
Error traceback:
func = <slot wrapper '__init__' of 'object' objects>
signature = <Signature (self, /, *args, **kwargs)>
first_arg = <[AttributeError("'ImportInput' object has no attribute 'transaction_id'") raised in repr()] ImportInput object at 0x7f2a2b2b3ac8>
data = {'self': <[AttributeError("'ImportInput' object has no attribute 'transaction_id'") raised in repr()] ImportInput object at 0x7f2a2b2b3ac8>, 'transaction_id': 'asdioansuiodb123asd'}
skip_superfluous = True, convert_types = False
def _unpack_dict(func: WrappedOrFunc, # pylint: disable=too-many-arguments
signature: inspect.Signature,
first_arg: Any,
data: Dict[str, Any],
skip_superfluous: bool,
convert_types: bool) -> Any:
"""Constructs an object in a type-safe way from a dictionary."""
assert _is_dict(data), 'Argument data needs to be a dictionary.'
ctor_params: Dict[str, Any] = {}
if not skip_superfluous:
param_names = [param.name for param in signature.parameters.values()]
argument_names = data.keys()
superfluous = set(argument_names) - set(param_names)
if superfluous:
raise TypeError(f'Superfluous parameters in call: {superfluous}')
parameter_values = list(signature.parameters.values())
if first_arg is not None:
parameter_values = parameter_values[1:]
for param in parameter_values:
if param.kind != inspect.Parameter.POSITIONAL_OR_KEYWORD:
> raise TypeError('Only parameters of kind POSITIONAL_OR_KEYWORD '
'supported in target functions.')
E TypeError: Only parameters of kind POSITIONAL_OR_KEYWORD supported in target functions.
Hi!
We had a confusing encounter today while using type_checked_constructor
with convert=True
.
A minimal reproducible example is the following:
from typing import NamedTuple, Optional
from undictify import type_checked_constructor
@type_checked_constructor(convert=True)
class Example(NamedTuple):
option = Optional[bool]
if __name__ == "__main__":
example = Example(option="False")
assert example.option is True
This is due to Python casting any string to True
if it's not empty, but it comes out as surprising if, as in our case, you are parsing a dictionary of GET parameters, that come in as strings.
What would you suggest as the best course of action? Is this something you might want to address in undictify or should we build a thin conversion layer just for booleans before feeding data into a class decorated with undictify's type_checked_constructor?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.