Cannot get DerivationPath serialised from root anymore about rust-bitcoin

[apoelstra]

@e1a0a0ea can you cite any part of the BIP that claims that the m/ prefix is part of a valid BIP32 path? I am pretty sure it is not. The BIP confusingly uses the prefix in its test vectors, but says in the flavor text that this is a placeholder for the actual fingerprint of the master key. IIRC it does not mention the prefix anywhere else.

BIP lawyering aside, we will probably add support for putting the m/ prefix back in, because it is used by HWI which is a major part of the ecosystem; see #2836 which is a dupe of this issue. Maybe we will use the # alternate formatter or something.

But @Kixunil is correct that there's an easy workaround here at least.

[Kixunil]

Apologies for causing a bunch of turnover on what I thought was a small ™️ change

No worries, better resolve it before 1.0 ;)

I think the confusion comes from the fact that BIP32 doesn't define a serialization for paths at all.

Indeed! I took a look independently before reading your comment and came to the same conclusion that it never says there is a format for the path.

Serialization for paths is defined in BIP380,

Would be nice to support that. Though I personally don't like that the format includes fingerprint.

or at the very least express paths using the method described in BIP380

I disagree because of the annoying fingerprint issue. Not only is it long but I know of at least one project where two signers use the same path but different keys. Further we can't store it in an array so it complicates our code and probably has more implications. I'd much rather see a different standard.

So do you acknowledge that before #2451 DerivationPath was getting serialised to BIP32's shorten notation but it isn't anymore?

It wasn't. That shortened notation uses the weird _H symbol for hardened paths which I don't think we can even produce in UTF-8 (I use GitHub HTML formatting here.) The previous serialization used ' or h depending on alternate formatting. (Oh, actually, this prevents us from printing the leading m/. 🤦‍♂️ Well, one more reason to not add weird functionality.)

So in conclusion, there never was any standard for serializing the array as UTF-8 and it doesn't look like there's going to be one, so we might as well pick the most sensible option which I think the current serialization almost is (I'd prefer to not have both ' and h). For BIP380 we need a different struct that also holds the fingerprint.

[Kixunil]

Serializing with m/ was against the BIP and thus wrong. There's software that still requires it, so just write println!("m/{}", derivation_path) in such cases. No changes needed to this library.

[e1a0a0ea]

No. What is against the BIP is forcing the serialisation with m/. Preventing any serialisation with it is also against the BIP.

[e1a0a0ea]

Well, in the Key tree section, it is stated that to shorten notation, we will write CKDpriv(CKDpriv(CKDpriv(m,3H),2),5) as m/3H/2/5. Equivalently for public keys, we write CKDpub(CKDpub(CKDpub(M,3),2),5) as M/3/2/5

Also, this picture uses the m/ prefix to explicit that the derivation is from the master node.

Sorry for the duplicate

[apoelstra]

To me, the phrase "to shorten notation" explicitly says "this is a shortcut and not valid notation".

The picture is more suggestive.

In any case, IMO we should just support it because HWI does. We can maybe file issues against the BIP and/or HWI complaining about it but for now we should support the de-facto standard and not the standard that we interpret from a confusingly-worded BIP.

[josibake]

Apologies for causing a bunch of turnover on what I thought was a small ™️ change (something something no good deed unpunished).

BIP lawyering aside, we will probably add support for putting the m/ prefix back in, because it is used by HWI which is a major part of the ecosystem; see #2836 which is a dupe of this issue. Maybe we will use the # alternate formatter or something.

For more context, I think the confusion comes from the fact that BIP32 doesn't define a serialization for paths at all. The only serialization defined in the BIP is for the extended keys.

Serialization for paths is defined in BIP380, where they are always represented as [8 hex characters for the root fingerprint]/NUMh/....

From a pragmatic point of view, I can sympathize with supporting the m/ prefix if it is still widely used, albeit incorrectly (or more precisely, wallets can display paths however they want, but this is in no way a standard defined by the BIP). I think the most correct thing to do is encourage projects to upgrade to using descriptors, or at the very least express paths using the method described in BIP380, i.e., [fingerprint]/num/num/etc.

[josibake]

No. What is against the BIP is forcing the serialisation with m/. Preventing any serialisation with it is also against the BIP.

At the risk of being overly pedantic, this is not true. The only serialization defined in the BIP is for extended keys, paths are not mentioned at all. Also, if the BIP were to define a serialization for paths and specified to use the character m, I'd argue this is a bad serialization scheme:

1. It is unclear if m refers to an xpub or xpriv, whereas BIP32 uses a mix of m and M
2. m is absolutely useless to the wallet unless they also have the root it refers to (either xpub or xpriv)
3. Furthermore, if a wallet has multiple xpubs/xprivs, they have no way of knowing which xpub / xpriv m refers to

All of these problems are addressed with descriptors, where paths are [fingerprint]/NUM/NUMh etc.

[e1a0a0ea]

@apoelstra when rust-bitcoin serializes a DerivationPath, isn't it serialized to the shorten notation?

But anyway, i see your point. The fact is I've seen this in many softwares so I guess i'm not the only one to be confused by this. So we should definitely either keep supporting this or update the BIP to remove the confusion

@josibake I agree descriptors express better the origin of a DerivationPath

[apoelstra]

@apoelstra when rust-bitcoin serializes a DerivationPath, isn't it serialized to the shorten notation?

No, not since #2451 which you linked in the OP.

[e1a0a0ea]

So do you acknowledge that before #2451 DerivationPath was getting serialised to BIP32's shorten notation but it isn't anymore?

[apoelstra]

Yes.