Support for serde::{Serialize, Deserialize} about borsh-rs

[dsxmachina]

Nice - thanks for the clarification @frol .
I will give it a try next month - and can hopefully present some results.

[preston-evans98]

Hey @dsxmachina (and FYI @frol), I looked at this in some depth a few months ago. Sadly, borsh is not compatible with serde's data model right now. The root cause of the issue is that Serde doesn't have a PartialOrd bound on keys for its map types, but borsh requires maps to be sorted during serialization to ensure that serialization is canonical. AFAICT, there are two ways you can try to work around this issue:

1. Sort maps based on the serialized values of the keys. This is safe, but its behavior will diverge from borsh's in many cases. (Integers are a good example; borsh uses little-endian, so the sorted order of serialized integers is completely different from numerical order). If all you care about is having a serde-compatible canonical serializer, you can take this approach though. If this is the route you take, you may want to look at the bcs library from Diem.
2. Use auto(de)ref specialization to try stay bitwise compatible with borsh within the set of types that borsh already serializes while also satisfying the compiler in the general case where keys are not comparable. (In the general case, you can either panic or try to fall back to comparing the serialized representation - both designs have tradeoffs, but that's a different discussion). Unfortunately, this design is extremely brittle. Autoref specialization only works with arguments to macros, not functions. And it doesn't work at all for nested structures. So, this approach can fail silently in a bunch of common cases, including if a user ever tries to serialize a nested map.

[dsxmachina]

@preston-evans98 thanks for looking into this.

@frol Maybe its not a bad idea to mention this under the "Feature" section in one short sentence.
You already talk about serde there and say that you opted out for serde for two features - then you name the features.

[frol]

Well, avoiding serde was a design decision:

Serde requires to use its abstraction layer that used to be (yet to be confirmed these days) a significant contributor to the compiled code size and performance degradation. Also, when considering serde adaptors, one should keep consistency in mind (a bijective mapping between objects and their binary representations; there should be no two binary representations that deserialize into the same object; this is extremely useful for applications that use binary representation to compute hash).

@dsxmachina If you want to give it a try, it would be interesting to see how far this experiment can go and whether the old beliefs are still applicable these days.

[dsxmachina]

Yes, this was what I was aiming at. I know you opted out of serde for performance reasons (and to enable the fancy 'init' feature) - but for me it would be okay to sacrifice performance over compatibility (and I don't know if code / binary size is an issue outside of the microcontroller world).

The question was more a "is it possible in principle" or "is it a stupid idea, because the bijective mapping will never work reliably with serde".
The latter is still not clear to me - would it be possible ? I think serde guarantees you that the serializers will always be called in the order that the struct fields are defined - does this do it?

[frol]

@dsxmachina It should work in principle, but the devil is in details, and we don’t have time/priority to dive into it, so I hope to hear from you or anyone else willing to jump on it if this is going to work.

[frol]

@preston-evans98 Thank you for diving into it and providing a detailed report! I will close this issue, but we may keep the conversation going and once it will become actionable, we may re-open it.