Test the `Encodable` impl for `Vec<T>` where `T: Encodable` about rust-bitcoin

[Kixunil]

Here's an idea: when push_decode is in we add the generic impl but try to downcast and simply return the bytes directly if the type is a slice (notice that push_decode has slice/iterator encoders). This is zero cost when the type is statically known.

Note that I feel uneasy about saying that all vectors in Bitcoin are and will be always encoded as varint-prefixed. Especially because it already isn't true for the sequence of Witnesses in segwit-encoded transaction.

[Kixunil]

If we restrict ourselves to a specific layout of what's logically a sequence then yeah, it's not an example. I tend to think about layouts separately from concepts and conceptually a sequence is best represented as Iterator - that's what I did in push_decode PR with a special exception for non-length-prefixed.

Anyway, not a big deal I guess.

[Kixunil]

Concept NACK, several reasons:

• conflicts with push_decode
• the impl probably kind of shouldn't exist (to be precise IterEncoder exists but Encode for Vec<T> probably shouldn't)
• we don't need to test that much - if each individual item is encoded correctly (and we test that) then any test that uses various lengths of arrays of some sufficiently distinct items (e.g. ints going 1, 2, 3, 4...) is enough because the serialization doesn't interact with how exactly an item is serialized (the distinct items requirement is only to detect stupid cases like repeating the first element n times)

Oh, and if coverage complains about it, it's probably just idiotic. It should see it as one line in impl not tested which doesn't matter.

[apoelstra]

There is no generic Vec<T> impl. We had to remove it because we couldn't specialize on Vec<u8> which is the overwhelmingly most common case and also the one which can be sped up significantly when we're not trying to be generic.

Having said that, agree with you that we should wait on push_decode to get this in to avoid conflicts.

But afterward we might as well stick these tests in there if it makes the coverage analysis happier. Yes, they are pointless, but they will run very quickly and nobody will ever have to look at them again.

[Kixunil]

I mean, if we're going to have an impl then we should probably have pub struct Inputs(Vec<TxIn>) & co to be explicit. I had another reason for it: allowing inherent methods. Though I don't remember which methods I thought were useful.

[apoelstra]

If there were inherent impls I think this could be worth doing. But otherwise it feels like we're just obfuscating the types.

But honestly I wish we did have Vec<T> for all T because then it would be clear that there's one way to consensus-encode vectors of all objects, and that we're not deviating from that somehow. But we can't, for efficiency reasons.

[dpc]

Note that I feel uneasy about saying that all vectors in Bitcoin are and will be always encoded as varint-prefixed. Especially because it already isn't true for the sequence of Witnesses in segwit-encoded transaction.

If they're encoded different they can use a different struct, it should be rare enough. And knowing that Vec<T> is always encoded certain way makes reasoning about things easier.

[apoelstra]

Especially because it already isn't true for the sequence of Witnesses in segwit-encoded transaction.

I'm confused. Witnesses are encoded as a Vec<Vec<u8>> in the "generic" way. This used to be how we encoded them a long time ago before the Witness type. We even had an Encodable impl for this type.

[Kixunil]

@apoelstra In transaction it would be Vec<Vec<Vec<u8>>> because there is a list of witnesses. And that list doesn't have a length prefixed - it's taken from the length of the inputs. This caused me quite a bit of refactoring to support when I was writing the code.

@dpc maybe, but it can get hairy with all the macros assuming Encode does the right thing. Though it doesn't apply in case of witnesses since that is even more complicated by being in inputs rather than separate.

Anyway, I'm not too much against it, just flagging it as potential issue.

[apoelstra]

I've never thought of it as a Vec<Vec<Vec<u8>>>. Aside from not being serialized like one, each witness is associated with a single input and is conceptually part of TxIn. If the number of witnesses differed from the number of inputs then the transaction would be invalid. So encoding as a Vec<Vec<Vec>> would increase the transaction size while allowing invalid encodings, and representing witnesses as a Vec<Vec<Vec>> (either in code or in your head) allows invalid objects to be represented.

Yes, this makes TxIn's encoding "non-local" which is pretty annoying, but it isn't an example of a vector that's encoded differently from other vectors.

[dpc]

Here's an idea: when push_decode is in we add the generic impl but try to downcast and simply return the bytes directly if the type is a slice (notice that push_decode has slice/iterator encoders). This is zero cost when the type is statically known.

This comment inspired me to do: fedimint/fedimint#4145