Comments (9)
This is something I would definitely like to support, since getting an error somewhere deep can be very annoying indeed. Need to think about it.
from cattrs.
We could copy a few ideas from jsonschema
, and potentially yield errors iteratively? Or maybe that's not really in scope, since cattrs
need to return the new result.
But have a look at their ValidationError, there's a few fields there that we could potentially use.
from cattrs.
This is probably the next big feature I work on :)
from cattrs.
Here is possibly quite crazy idea how to report an error incl. path within input data leading to the failure.
Requirements:
- initially focus only on
structure
and assume input data in form ofdict
- allow reporting path to input data element causing raised conversion error
- keep required changes to code (structure_hook functions) to bare minimum
- tolerate structure_hook implementation not implementing new approach (possibly at cost of loosing part of path information)
- run fast, try to avoid any extra operations during happy scenario
Concept:
- path has form of list of
__getitem__
arguments, e.g.["oak", 1]
data["oak"][1]
- no need to cover non-iterable data types
- focus on iterables. Store current position of iteration in local variable with agreed name
cattrs_i
. This is the only required change tostructure_hook
function implementation. - all path detection to be done within
cattr.structure
function- detection is done by traversing traceback stack, inspecting local variables and collecting all values of
cattrs_i
variables in resulting path list. - store the path in
.path
exception property and raise the catched exception
- detection is done by traversing traceback stack, inspecting local variables and collecting all values of
Here is code to demonstrate how to detect the path from an exception raised in deeply nested call. If you store the code into test_path_detection.py
, it shall be executable using pytest
(expecting python 3.6+).
def int_structure_hook(val, dtype):
return int(val)
def list_structure_hook(lst, dtype):
return [int_structure_hook(itm, int) for cattrs_i, itm in enumerate(lst)]
def dict_structure_hook(dct, dtype):
return [list_structure_hook(val, list) for cattrs_i, val in dct.items()]
def structure(val):
try:
return dict_structure_hook(val, dict)
except ValueError as exc:
path = []
tb = exc.__traceback__
while tb:
path_elm = tb.tb_frame.f_locals.get("cattrs_i")
if path_elm:
path.append(path_elm)
tb = tb.tb_next
exc.path = path
raise exc
def test_it():
try:
res = structure({"oak": [1, "0aa", 3], "birch": [9, 2, 0]})
print(f"Happy result is: {res}")
except ValueError as exc:
print(f"Path {exc.path}: has problem: {exc}")
When called:
$ pytest test_path_detection.py -sv
the printed output related to reported path is
Path ['oak', 1]: has problem: invalid literal for int() with base 10: '0aa'
What do you think of that? No perfect results, but something, what helps navigating close to source of problem in many cases. Definitely would require (small) modifications in existing converters.
from cattrs.
@Tinche take your time, it is not an easy problem.
Here is alternative method: pass path
via explicit argument to conversion function:
"""alternative passing path context via argument `path`
Converters have singature: func(val, dtype, *path)
where `path` is the path to the current element (list of values)
When calling, one uses original `path` value with * and adds new selector to the end
fun(val, dtype, *path, index)
what results in extended `path` value within the deeper function.
"""
def int_structure_hook(val, dtype, *path):
return int(val)
def list_structure_hook(lst, dtype, *path):
return [int_structure_hook(itm, int, *path, i) for i, itm in enumerate(lst)]
def dict_structure_hook(dct, dtype, *path):
return [list_structure_hook(val, list, *path, key) for key, val in dct.items()]
def structure(val, dtype):
try:
return dict_structure_hook(val, dict)
except ValueError as exc:
path = []
tb = exc.__traceback__
while tb:
deeper_path = tb.tb_frame.f_locals.get("path")
if deeper_path:
path = deeper_path
tb = tb.tb_next
exc.path = path
raise exc
def test_it():
try:
res = structure({"oak": [1, "0aa", 3], "birch": [9, 2, 0]}, dict)
print(f"Happy result is: {res}")
except ValueError as exc:
print(f"Path {exc.path}: has problem: {exc}")
assert exc.args[0] == "invalid literal for int() with base 10: '0aa'"
assert isinstance(exc, ValueError)
assert exc.path == ("oak", 1)
To avoid confusion with intermediate functions using path
argument, traversing __traceback__
may check, that givel locals are within function which is registered at converter.
from cattrs.
Incidentally, one of the hardest things to debug is when you have a NoneType that can't be converted into whatever the expected type is. Without a path, currently there's no way to even guess at which of the many null
s in your input it's failing on.
from cattrs.
Hey, sorry for kinda necro-post, but have there been any progress on this? It would really be very handy to have this feature :)
from cattrs.
Nice to hear it! Sorry for the question, but do you have any ETA for it?
from cattrs.
So there's https://catt.rs/en/stable/validation.html#transforming-exceptions-into-error-messages in the last release, 23.1.x.
I'm going to close this as complete, let's open new tickets for any desired improvements!
from cattrs.
Related Issues (20)
- JSON serialisation fails HOT 8
- concurrent.futures.ProcessPoolExecutor emits a very useless error message because the cattrs Exception cannot be pickled HOT 6
- support typing_extensions Literal HOT 4
- Unstructuring regression for optional generic types HOT 4
- Cattrs lib issue HOT 3
- When structuring, how to forbid type casting to base classes? HOT 2
- include_subclasses doesn't seem to work on members of a class HOT 1
- cattr vs. cattrs need documentation HOT 4
- Support TypeAliasType for structuring HOT 1
- Using class methods strategy for structuring Enums fails without clear info HOT 4
- Option to include `init=False` globally on a converter, not per class HOT 2
- KeyError when using configure_tagged_union with existing field HOT 3
- Structuring fails with PEP 695 aliases to Literal HOT 3
- Structure failed with `typing_extensions.Any` HOT 3
- Constructor selection for structuring HOT 7
- Tagged Union: How to do structure/unstructure hook(s) for particular member type? HOT 4
- A dictionary gets structured into a list without errors/warnings HOT 2
- Question: Correct use of converters HOT 3
- Documentation does not explain unstruct_collection_overrides' input HOT 4
- Detailed validation exception groups with hook errors HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cattrs.