Document serialization guarantees and drop serde from anything that isn't in Core RPC about rust-bitcoin

[apoelstra]

For now let's do #[serde(rename)].

I think when we're closer to 1.0 we should do a "break the world" serde release (and make very few other changes along the way) and see how much pushback we get over it, and possibly revert it. I'd prefer if we had a cleaner/more principled serde serialization before 1.0 ... but we may be stuck with it ... and in any case I don't want to break serde piecemeal over several versions.

[apoelstra]

"Drop serde from everything that isn't in Core RPC" feels a little strong. Though I guess almost everything -- WIF keys, xkeys, BIP32 paths, PSBTs, etc., all do appear in the RPC in some place.

As for the use of deny_unknown_fields, I could go either way on this. I don't want to force people to use bitcoin-core-rpc if they don't have to, but OTOH it's really not difficult to write wrapper structs yourself to handle the output of getrawtransaction and so on. As long as you only need a couple such structs. I'm tempted to just put deny_unknown_fields everywhere and accept breakage for people trying to use raw rust-bitcoin types directly with the RPC.

I note that we're already not compatible with Core's RPC for e.g. Transaction because our input/output/etc fields don't match the vin/vout fields that the Core RPC uses. So on reflection, I don't know that anybody could possibly be using raw rust-bitcoin types with the RPC. Other than the simple types (e.g. Address) which have uncontroversial string encodings.

For de/serialization from Core RPC I think we should provide an example that uses serde(from) and serde(into) to demonstrate making a dummy struct that mirrors the RPC output and letting serde do the conversion from the dummy struct into Transaction and vice-versa. And saying that you should probably use bitcoin-core-rpc crate if you want this done right.

Where does this leave us, serde-wise? I do want to keep the serde feature and have De/Serialize implemented on all the main types, because it's useful for e.g. dumping bitcoin data into json config files or whatever. And we've put a lot of work into e.g. Amount to make sure that it can be serde-deserialized in the broad range of formats that we see it used in different exchange APIs.

But yes, we need to clarify who our target is and come up with a coherent policy for what we have serde traits for.

[Kixunil]

I note that we're already not compatible with Core's RPC for e.g. Transaction because our input/output/etc fields don't match the vin/vout fields that the Core RPC uses. So on reflection, I don't know that anybody could possibly be using raw rust-bitcoin types with the RPC.

Oh, interesting. So we're in a pretty bad situation then. We can't sync them up with Core because that'd break someone and now we have just arbitrary field names. And I have no idea how many users actually use the serde serialization. Honestly maybe we should just change it into hex string/bytes since that format is practically immutable.

simple types (e.g. Address) which have uncontroversial string encodings.

Those I definitely want to keep.

And I fully agree that having serde is useful, great even but if we have weird, unstable serialization format it could be counterproductive.

[apoelstra]

Oh, interesting. So we're in a pretty bad situation then. We can't sync them up with Core because that'd break someone and now we have just arbitrary field names. And I have no idea how many users actually use the serde serialization. Honestly maybe we should just change it into hex string/bytes since that format is practically immutable.

It's tempting, but for example, I'm pretty sure I'm using the existing serialization in some obscure personal tax software (where I'm using the raw transactions, along with merkle proofs, as evidence to be produced in case of audits). Personally it wouldn't be a big deal for me to change this, but if I wasn't a developer of the library I'd be super annoyed.

I think though we should be able to add deny_unused_fields though because our existing format is ad-hoc, so it's hard to imagine anyone deserializing something in this format that didn't come from this library.

Perhaps we want to go through the library, remove all the derive Serialize/Deserialize (except maybe for some trivial cases), and move the serialization logic into a single "do not touch this" module. Though maybe the existing serde fixed-vector tests are sufficient for this, if we were to complete them.

[Kixunil]

OK, that sounds good. But what do we do about things like renaming input to inputs? Do we put #[serde(rename="input")] on it or just accept the breakage (because even in format plural looks better). Or do we create some sort of migration module?

[Kixunil]

Sounds good!

[tcharding]

Though maybe the existing serde fixed-vector tests are sufficient for this, if we were to complete them.

They were complete at the time I did them but by their very nature go stale easily, if we do all this we should try to enforce that regression tests exist on ever serde'd type in CI.