Comments (5)
No, that's accurate. Custom types supported by dec_hook
could result in cyclic behavior, but in general it's unlikely for the result of a decode
call to have any cycles. But code constructing these objects outside of decode
could still result in a cycle. The warnings are mostly to let users know "here be dragons" and to deter them from mucking with the gc
unless a benchmark shows it matters. That you can properly reason about cyclic object structures and python's GC implementation means you are probably capable of judging whether disabling it on these has consequences for your code :).
from msgspec.
python thinks it should track UUIDs even though they are immutable and only contain a couple underlying primitives
In CPython any type implemented in pure python is a GC type. Since uuid.UUID
objects aren't extension types (i.e. they're implemented in python) then they're automatically GC types. If uuid.UUID
types were implemented as extension types then you're correct, they wouldn't need to be a GC type.
One option if you'd rather disable GC on the type instead of globally - If you don't ever manipulate the UUIDs as uuids you might try annotating those fields as str
instead (possibly with a pattern regex for matching uuids if you're concerned about invalid uuids getting in). Strings are immutable non-GC types. That said, for large payloads the overhead of turning on/off the gc per decode
call should be minimal.
from msgspec.
Thanks for opening this! We were originally being a bit stricter than necessary here. The real limitation is types with gc=False
must be __slots__
classes, so any mixin type (like Generic
) must also define __slots__ = ()
. With #635 you should be able to set gc=False
on generic structs as well.
In [1]: from typing import Generic, TypeVar
In [2]: from msgspec import Struct
In [3]: P = TypeVar("P")
In [4]: class Demo(Struct, Generic[P], gc=False):
...: x: P
...: y: P
...:
In [5]: d = Demo(1, 1)
In [6]: import gc
In [7]: gc.is_tracked(d)
Out[7]: False
Standard note - messing with the gc
kwarg is considered "advanced usage", I trust you've read all the warnings in the docs before using it :).
from msgspec.
@jcrist thanks for the fix!
I have read the documentation on that, however, I'm confused on one point:
Why would any struct that participates in deserialization not be a good candidate for gc=False
? As we know, when deserializing JSON to a normal dict, it is impossible that that JSON is self-referencing. I.e., you can't have a thing inside itself simply because that is impossible to represent as JSON! So any reference cycles for any of these objects participating in deserialization would by nature have to be created manually in the post_init or subsequent stages. So as long as I am not "adding a thing to itself" post-init, and these structs originate from JSON, I should be totally safe for gc=False
.
Or am I missing something?
from msgspec.
@jcrist awesome! One thing I did notice during my benchmarks is that gc=False
is somewhat undermined by the presence of UUID
fields... for some reason python thinks it should track UUIDs even though they are immutable and only contain a couple underlying primitives. I tried to find something on how to "untrack" UUIDs... but was unsuccessful... so I ended up just disabling garbage collection altogether until all my objects are destroyed anyways by refcounting.
from msgspec.
Related Issues (20)
- Docs page on testing
- json schema generation - differences between pydantic and msgspec HOT 3
- Allow conversion to collection from generator HOT 2
- Porting guide for users coming from `orjson`
- Converting dicts into list with key-reuse HOT 3
- Collecting multiple validation/constraint errors at once HOT 1
- Allow `omit_defaults` to exclude fields when encoded value is `{}` (empty dict)
- Duplicate key detection
- Allow unknown tags, defaulting to tagged base
- Implementing optional bytes type for json. HOT 1
- Update annotation parsing to work with PEP 649 in Python 3.13
- `omit_defaults` does not omit tuples and frozensets HOT 2
- Field Alias Overrides in Subclasses Not Reflected in __struct_encode_fields__
- Convert builtin types to numpy HOT 1
- Is it possible to have the decoding of union of all subclasses of a struct
- Subclasses of frozen Structs causing mypy error: `Cannot inherit non-frozen dataclass from a frozen one` HOT 2
- Datetime without timezone are decoded as str with msgspec.msgpack. in 0.18.6 HOT 1
- Optional[Raw] results in an unexpected validation error at json.decode time HOT 5
- Struct import fails in pytest - Missing requirement argument 'attribute x' or unhashable type: 'StructClass'
- Generate FastAPI docs from msgspec.Struct json schema HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from msgspec.