Comments (4)
This is the currently intended (or at least not unintended) behavior. In my mind there are 3 main use cases for encode_into
, none of which would run into this issue:
- Encoding into a pre-allocated buffer. In this case
offset
is never needed.
buffer = bytearray(1024)
for msg in msgs:
enc.encode_into(msg, buffer)
socket.sendall(msg)
- Encoding into a pre-allocated buffer, leaving space for a prefix. This is the length-prefix framing example in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.
buffer = bytearray(64)
encoder.encode_into(msg, buffer, 4) # offset for prefix, but 4 is less than 64 and both are hardcoded sizes (not data dependent)
n = len(buffer) - 4
buffer[:4] = n.to_bytes(4, "big")
socket.sendall(buffer)
3. Accumulating several writes into the same buffer, perhaps with a delimiter. In this case you're always writing to the end of the buffer, so offset is -1
```python
buffer = bytearray()
for msg in msgs:
encoder.encode_into(msg, buffer, -1)
buffer.append("\n") # perhaps we're writing line-delimited json
That all said, I can see how the current behavior would be surprising, and would be open to changing it to automatically grow the buffer to support the specified offset
.
Out of curiousity, what was your use case that led to finding this issue?
from msgspec.
The use case was actually case 2.
Encoding into a pre-allocated buffer, leaving space for a prefix. This is the length-prefix framing example in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.
I just forgot to add the size on one of my buffers. Python has no problem doing x = bytearray()
without size, and decode_into
has no issue writing to this buffer. Just the right amount of coincidences happend and my encoded message could still be decoded (at first). It's only when I started to expand the tests that i suddenly got weird results, and messages that couldn't be decoded.
Since both just work, it was quite hard to pinpoint where my message got "corrupted". A mention about this in the docs would've saved me some time here.
For context i'm writing an ASGI RPC app (and client) using msgspec for serialisation.
from msgspec.
Hmmm, in that case maybe we should error instead if offset > len(buffer)
? That would have caught your issue and pointed you to a better usage pattern. On the other hand, making it work by expanding the buffer is also easy, and would have negligible performance impact (at least for reasonable offsets) since we'd have to grow the buffer anyway to write the message. Waffling a bit here on what the best UX is.
from msgspec.
I can see benefits for both options. Expanding the buffer will "Just work" and I can't really see a use case where you might not want that to happen, especially since encode_into
expands anyway when necessary.
On the other hand going by the zen of python:
Explicit is better than implicit.
and
Special cases aren't special enough to break the rules
If anyone ever has a use case where they don't want this expanding to happen, it might be hard to figure out what is going on.
from msgspec.
Related Issues (20)
- Support encoding any Enum value type
- Json Encoder: dataclass classes are encoded to empty dict
- Tighten typing of msgspec.json.schema
- Support numeric constraints for Decimal values
- Support decoding empty strings as unset for all field types
- Does msgspec have a concept of schema mode?
- Add an option to disable constraint checks
- Ability to add an example to a Struct as output to the JSON schema
- Unable to encode `lxml.etree._ElementUnicodeResult`
- fallback to dict for unknown type in tagged union? HOT 3
- Multiple types to same Struct HOT 1
- Encode field name
- msgspec.json.decode parsing a float value HOT 10
- 3.13 support HOT 1
- Converting an object / dict to a struct where the struct contains a Raw field fails with "Expected `any`, got ..."
- Incorrect decoding of 65-bit integer
- Is there a way to stop parsing long lists of JSON objects when an object with a certain property value is found
- Allow passing custom globals to use when converting dataclasses
- Positional-only arguments
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from msgspec.