Signing API redesign about rust-bitcoin

[tcharding]

not bitcoincore-rpc which has stalled updates to the last 5 or 6 rust-bitcoin releases for everyone who depends on it

I got added as maintainer to the rust-bitcoincore-rpc repo as well as publish perms on crates.io - so we should be right during releases from now on. (Assuming I don't go AWOL too ;)

[apoelstra]

This might be as good a time as ever to introduce testing against bitcoind in CI (regtest), I rekon we should have tests that create various chains of transactions, and verify that we did it all correctly. This would serve as good documentation as well as testing.

This could be quite difficult because it would create a circularity between the bitcoind crate and rust-bitcoin. (Though we could skip this and hack together a "spawn a bitcoind" script in CI.)

[Kixunil]

Most people I have seen in workshops (maybe @tcharding can chime in here) struggle with this additional magical parameter SigHashCache that pops up in between and unintuitive at first.

Interesting! Maybe if we just had a method on transaction .signing_builder() it would work? It does act a bit like a builder once you have it. :) I suspect people mostly search for sign method on Transaction.

The optimization will still work for sign_all APIs that sign/verify multiple inputs in the single function call. I feel we should be able to achieve a simple and efficient API for most wallets that mostly want sign_all method.

Oh, good point! That's pretty much what SighashCache::sign_all does and we could simply have:

impl Transaction {
    fn sign_all(&mut self, prevouts: &[impl AsRef<TxOut>], f: impl FnMut(SigningInput)) {
        SighashCache::with_prevouts(self, prevouts).sign_all(f)
    }
}

Which makes a lot of sense.

[apoelstra]

I like the idea of using "builder" terminology rather than "cache" terminology, and hanging it off of Transaction rather than making the user construct one and passing in a transaction reference. It's super unintuitive that signing would need a "cache" and that you should start the signing process with such a thing.

We should, of course, make sure the advanced process is still possible and allow creating the cache with a generic T: Borrow etc. But it shouldn't be foregrounded.

[apoelstra]

It's hard for me to tell what's going on without seeing how we handle non-Taproot inputs. What do the sign_all or inputs iterators yield in this case?

Also your second impl block is missing a type so we can't tell what Self is.

[Kixunil]

It's hard for me to tell what's going on without seeing how we handle non-Taproot inputs.

Similarly, the Input handle will have methods for signing them. Which reminds me I forgot about them in #2347

Also your second impl block is missing a type so we can't tell what Self is.

Fixed.

[apoelstra]

So, the Input handle has methods for producing both Taproot and non-Taproot signatures? How does the user ensure that they're using the correct method?

[Kixunil]

How does the user ensure that they're using the correct method?

Just like now, they have to know it out of band. We might add even more machinery to put that into extra but maybe it's too much. I can try to think about it though.

[apoelstra]

Ok. So, concept ACK adding these iterator methods which eliminate annoying IndexOutOfRange errors (and having the associated Input type to facilitate this). I kinda think Input should be an enum over all the different input types so that users can match/destructure and then be forced to use the correct signing methods and signature types.

I'm uncertain about having some signing methods update the transaction ... you can do this with p2pk(h), p2wpkh and p2tr keyspends, but not for any other type of spends, so I think the resulting API is going to feel inconsistent.

[tcharding]

This might be as good a time as ever to introduce testing against bitcoind in CI (regtest), I rekon we should have tests that create various chains of transactions, and verify that we did it all correctly. This would serve as good documentation as well as testing.

For example

1. Take some inputs and create a 2-of-3 multisig output
2. Sign the multisig output and send it to some other set of outputs (spend, change etc.)
3. Do key path spend and script path spend of outputs created in (2)

Try to cover various output types (segwit v0 and taproot).

[tcharding]

Good point, we could put the testing in the new psbt crate? It is going to be pretty tightly coupled with rust-bitcoin anyways and I was considering maintaining a constant dev branch to track rust-bitcoin master and test stuff as we do it.

[junderw]

In BitcoinJS we had a hard time designing the signing API in regards to async/await.

Many users had use cases for signing on hardware wallets without blocking the event loop (JS).

It seems like the async use case is pretty strong for signing in particular, and I wonder how wallets coded in Rust would handle that with this API? Spawning threads seems to be the only option for now.

Would something using the async_trait crate be acceptable here? Or would we want to wait until we bump MSRV above 1.75 or some other stable version that adds enough functionality to async traits (dynamic dispatch is a big one I think it's missing rn)

Thoughts?

Edit: Perhaps this is a bigger discussion, we can discuss it here: #2407

[Kixunil]

it would create a circularity between the bitcoind crate and rust-bitcoin

Not after we split primitives out - bitcoincore-rpc will only depend on primitives, and sighash will dev-depend on primitives (I'm thinking that bitcoin-sighash crate could be appropriate.)

Further even if we wanted to do funny things like testing serialization against bitcoind, it's a dev-dependency which might be allowed in integration tests and we can depend on an older version anyway, which is allowed. Maybe we'd have to bump the version to 1.x.$(last + 1)-dev right after release but that may be a good thing.

@junderw so first of all, I think that the signing logic should be moved to HWW entirely - there's no point in having HWW if it blindly accepts a hash which it signs without any further verification and if you have the code capable of verifying then it already can produce the hash to sign so this logic is not needed in application at all. The only reason to have the keys stored separately is if you're thinking that "sure the attacker will steal this but I can continue using my previous seed and don't have to rotate". Which apparently is something people like.

But I'd still like to be async-friendly and if we get rid of all closures it already is. So I really need to look into iterator and even if we can't impl Iterator we can do the streaming version (and we don't really need GAT/streaming iterator, being able to write while let Some(input) = inputs.next() is enough).

[junderw]

I think that the signing logic should be moved to HWW entirely

In the ideal situation, that would be best. But I can tell from personal experience that some of these HSM vendors really make it hard to run custom binaries on their machines, and some companies require certain HSMs for regulation compliance etc.

Obviously, if you're a Trezor or a Ledger and you're building your own boards from the ground up then I fully agree.

Maybe async is something for a potential 2.0 many years down the line once stable Rust figures out async traits a bit more and that becomes an easily MSRV-able version... but I was just bringing it up as something to possibly consider when designing the non-async API. (ie. making the API easier to expand into having async counterparts later on etc.)

[Kixunil]

and some companies require certain HSMs for regulation compliance

Not sure if I should laugh at them or curse them... 😆 but sure, I can have no problem if people are bypassing that crap and I'll be happy to help them.

Maybe async is something for a potential 2.0 many years down the line once stable Rust figures out async traits

Not really, this is perfectly achievable in 1.0 and something we actually aim to provide.

[apoelstra]

Not after we split primitives out - bitcoincore-rpc will only depend on primitives, and sighash will dev-depend on primitives (I'm thinking that bitcoin-sighash crate could be appropriate.)

We'd depend on bitcoind whose maintainer doesn't go AWOL for months at a time, not bitcoincore-rpc which has stalled updates to the last 5 or 6 rust-bitcoin releases for everyone who depends on it :). But yes, your point is the same.

I think we can evade a literal dependency circle by using a separate-crate integration test, as you hint at. This would work even without crate smashing. But we still have a "conceptual" circularity .. where if we update rust-bitcoin, our integration tests will be broken until bitcoind updates, which it can't do until we release rust-bitcoin.

Maybe crate smashing will get us out of this. I hope so. I think we're still uncertain whether bitcoin-primitives can actually work as an independent crate or whether we run into too many coherency issues.

@junderw

In the ideal situation, that would be best. But I can tell from personal experience that some of these HSM vendors really make it hard to run custom binaries on their machines, and some companies require certain HSMs for regulation compliance etc.

Lol!! If HSMs are signing blind hashes to comply with some sort of regulation that requires HSMs but allows that, they can write their own APIs for it.

But we should move this discussion to the async discussion #2407 that you opened (or Kix's isssue #2409 if there are specific API pain points we need to address).

[Kixunil]

We'd depend on bitcoind whose maintainer doesn't go AWOL for months at a time, not bitcoincore-rpc

bitcoind depends on bitcoincore-rpc, not directly on bitcoin.

"conceptual" circularity .. where if we update rust-bitcoin, our integration tests will be broken until bitcoind updates, which it can't do until we release rust-bitcoin.

This is only a problem if we break the API which we will eventually stop doing.

[apoelstra]

Oh, excellent :) So maybe this fear is not an issue.

[sanket1729]

@Kixunil, Still parsing through the post, but this is a bit heavy for me after months of absence :) . One point I think we should investigate if we really need the efficiency gains from SigHashCache.

I am in part responsible for suggesting/implementing here, but I think we should revisit now if the efficiency gains are worth the complexity/API mess. We base this primarily from bitcoin core verification logic.

• Signing is already an expensive crypto EC operation. Signing schnorr takes 40-50 micros, hashes take 50ns; we are not saving a lot by caching things here. It makes sense for bitcoin core, because they use caching for verification for all inputs in the same transaction and verification is a frequent operation that needs to be fast. Signing on the other hand can be slow, infrequent, often is clubbed up with other expensive steps like bip32 derivations and waiting for network calls to external signer.
• We still use SIghashCache internally for APIs that sign multiple inputs at once. However, the APIs that want fine grain control over APIs they pay the extra cost of hashing the same thing again.

[Kixunil]

I think one could still conceivably use it for verification (e.g. in Firefish we do verify signatures but we don't need it to be crazy fast). And it feels like it'd be a shame to throw away this working optimization. Also I don't think it's cache that's causing most API issues. It's the inputs to the functions being dynamically checked, so there's a bunch of possible errors and then whatever happens after signing - putting the data into witness etc.

We've improved some things but there's more stuff to do.

[sanket1729]

@Kixunil

Most people I have seen in workshops (maybe @tcharding can chime in here) struggle with this additional magical parameter SigHashCache that pops up in between and unintuitive at first.

And it feels like it'd be a shame to throw away this working optimization.

The optimization will still work for sign_all APIs that sign/verify multiple inputs in the single function call. I feel we should be able to achieve a simple and efficient API for most wallets that mostly want sign_all method. The advanced users can deal with cache APIs.

[Kixunil]

make sure the advanced process is still possible and allow creating the cache with a generic T: Borrow

Of course! And we can obviously have a method on ExtendedTransaction too which will use the other method so it uses its own prevouts.

One issue though is that inherent method interferes with stabilization of Transaction before SighashCache (do we actually want to rename it to SigningBuilder?) I think we could just wait with adding it until SighashCache stabilizes at which point we can just invert the dependencies. This is not a problem since we use a trait anyway.