Investigate libsecp256k1 about bchd

[romainPellerin]

@cpacia I was underwater for the past few days building https://ramen.chopsticks.cash that is based on ECMH signatures, I will have a look to this in the next days.

[cpacia]

Great thanks

[swdee]

Had a closer look at the CGO docs and CGO is enabled by default for native builds, but is disabled when cross compiling. On native builds one can disable CGO use with the environment variable, eg: CGO_ENABLED=0 go build.

Cross compiling with CGO is doable but complicates the build process, although there are some toolchain packages that make it easier. With build tags we could limit CGO usage to be exclusive to linux.

Even if there are 1 million transactions per block with median transaction size at 225 bytes (~214MB blocks), we are looking at 43 seconds verification time via bchec and 15 seconds via libsecp256k1. There are plenty of other bottlenecks to address to get to those levels, considering BU's giganet tests maxed out at around 22MB blocks.

As faster verification times really only benefit mining I would favour keeping things simple and staying in pure go for now considering there isn't any demand for it and I see your recent work on a go implementation of ECMH which further negates the benefits of getting that functionality via cgo/libsecp256k1.

[DesWurstes]

This might help you: https://github.com/ethereum/go-ethereum/tree/master/crypto/secp256k1

I can help you make the cgo code there become compatible with the latest libsecp256k1, there are a few little incompatible changes.

[cpacia]

@DesWurstes that would be great. I have taken a brief look at how ethereum does it and noticed it's not compatible. I would also be nice if we could include Thomas' ECMH module https://github.com/tomasvdw/secp256k1/tree/multiset/src/modules/multiset

[zquestz]

I would be willing to move to the faster libsecp256k1 library from gocoin. I am hesitant to add a library that requires c bindings, as they complicate the build process. For instance, how do we make sure the standard go get -u github.com/gcash/bchd call continues to work fine?

[cpacia]

I think go get will still work but I think the catch is the c code is recompiled on each build. Maybe someone who knows more about cgo would chime in.

[romainPellerin]

We just released this lib https://github.com/eminently/go-crypto which is basically go bindings to ECMH (multiset) from Bitcoin ABC secp256k1 C++ lib, because we have plans to compare CTOR and NTOR blocks within chopsticks.cash.

So we could merge it in bchd with pleasure.

The question is: what needs to be bound to this lib within bchd?

@cpacia @zquestz

[cpacia]

Great work @romainPellerin. Just looking at it briefly tho the signing and verifying uses the pubkey recoverable format which ethereum uses. We'll need the other format.

[romainPellerin]

Ok I can add what's missing, let me know the list of features required.

@cpacia here is a summary of our conversation with @zquestz on chopsticks.cash Telegram (you have been invited to join it):

• @zquestz told me that you have a script that will help us compare the perfs of gocoins and these c++ bindings we released, could be a good starting point to verify that bchd will benefit from it;
• then we need to see if these bindings can cover all bchd needs or if we need to add more (echd for example);
• and finally, we need to estimate the refactoring effort (code impact).

[cpacia]

I think the functions we need are in https://github.com/eminently/go-crypto/blob/master/secp256k1/libsecp256k1/src/ecdsa.h

At least sign and verify, but maybe the other two as well. Though they aren't as important.

[swdee]

Overview

Had a look at this one today and these are my findings.

• gocoin has a golang implementation of secp256k1 which is an implementation of bitcoin cores libsecp256k1 at https://github.com/piotrnar/gocoin/tree/master/lib/secp256k1

• gocoin has a CGO wrapper to libsecp256k1 that implements only the signature Verify() function. This makes sense they have not created bindings to other functions as the majority of performance benefit seen would be in signature verification (as opposed to signing), particularly at time of syncing and block processing. By default the CGO wrapper is disabled so go get works as normal, the CGO wrapper however can be activated by copying a go source file into the project root and builds providing libsecp256k1 has been compiled and installed in the system library directory ie: /usr/local/(lib|include).

• Ethereum has CGO bindings to libsecp256k1 but they also have a build system in place to create the binaries and check the build environment meets all dependencies.

• Compiling libsecp256k1 is quite quick, around 25 seconds on a 2 core VM.

• There is also a set of CGO bindings here https://github.com/copernet/secp256k1-go/tree/master/secp256k1 that are quite extensive in terms of exposing the C functions. They seem to be created 2 years ago with one recent bug fix. They do not expose the Multiset/ECMH capabilities.

Benchmarks

First tests were to run gocoins benchmark scripts at github.com/piotrnar/gocoin/lib/others/cgo/ec_bench with the following results. Note that the results vary depending on CPU speed however they do show the performance relative to each implementation. These benchmarks time how long it takes to perform a Verify() operation.

Implementation	Time per Verify() operation	Benchmark Function name Below
Gocoin go/secp256k1	35 us	BenchmarkGoCoinGo
Gocoin CGO/libsecp256k1	15 us	BenchmarkGoCoinSipa
Gocoin CGO/OpenSSL	101 us	n/a
Bchd go/bchec	43 us	BenchmarkBchec

As @cpacia notes on the opening of this issue, gocoins secp256k1 implementation is slightly faster than bchec.

I created some go like benchmarks for each.

$ go test -bench=.
goos: linux
goarch: amd64
pkg: dev.kpt.bz/tests/cryptobench/benchall
BenchmarkBchec-8                    3000            400092 ns/op           0.00 MB/s        4002 B/op         78 allocs/op
BenchmarkGoCoinSipa-8              10000            113769 ns/op           0.01 MB/s           0 B/op          0 allocs/op
BenchmarkGoCoinGo-8                 5000            337216 ns/op           0.00 MB/s       10805 B/op        168 allocs/op
PASS

The benchmark performance looks good so I integrated gocoin's CGO wrapper into the bchec package on a branch here https://github.com/swdee/bchd/tree/gocoin-sipa

Building bchd with libsecp256k1

bchd can be built on this branch following these instructions.

1. Download libsecp256k1 from https://github.com/bitcoin-core/secp256k1

wget https://github.com/bitcoin-core/secp256k1/archive/master.zip
unzip master.zip
cd secp256k1-master/

2. Build libsecp256k1 per vendor instructions.

./autogen.sh
./configure
make
sudo make install 

3. Get forked bchd repository and checkout branch

go get github.com/swdee/bchd
cd $GOPATH/src/github.com/swdee/bchd/
git checkout gocoin-sipa

4. Test CGO and libsecp256k1 build ok. Note use go v1.10.1 or higher, there is a bug with CGO on v1.9.4. A successful test should output with PASS.

cd $GOPATH/src/github.com/swdee/bchd/bchec/cgo/sipasec/
go test 

5. Activate CGO bindings for building into bchd

cp $GOPATH/src/github.com/swdee/bchd/bchec/cgo/use_sipasec.go $GOPATH/src/github.com/swdee/bchd/

6. Build bchd binary

cd $GOPATH/src/github.com/swdee/bchd
go build

7. bchd will now be built and when started it outputs to stdout indicating libsecp256k1.a is being used.

$ ./bchd
Using libsecp256k1.a by sipa for EC_Verify 
2018-11-26 09:54:28.893 [INF] BCHD: Version 0.13.0-beta 

bhcd Testing

The next step was to launch two VM's (2 CPU cores/4GB RAM) and run bchd and bchd/libsecp256k1 on the separate instances to test syncing time from the genesis block. Unfortunately real world testing was disappointing as there was no improvement to sync speed at all by having faster signature verification. The network and peer interaction appears to be the bottleneck with regards to block processing speed so any improvement in signature verification can't be seen.

@cpacia As you implemented the UTXO cache work you would have a better idea of the bottlenecks in the blockchain verification code flow, hence what are your thoughts here as this optimisation does not appear to offer meaningful value?

[swdee]

Using a pure calculation on performance gained when syncing, a typical log entry looks like;

[INF] SYNC: Processed 23 blocks in the last 10.88s (16038 transactions, height 447270, 2015-06-07 21:37:46 +0000 UTC)

Given a 43 microsecond processing type of bchec per verify() operation and 15 us for libsecp256k1 we get the following performance difference.

bchec: (16038 transactions * 43us / 1000) = 689ms total processing time
libsecp256k1 : (16038 transactions * 15us / 1000) = 240ms total processing time

[cpacia]

Thanks for doing all of that @swdee. That's a lot of good work.

As for the sync speed. Since the most recent checkpoint is Nov 15, it will skip signature checks before that date. Which means a node using libsecp256k1 should take around the same amount of time compared to one which doesn't use it.

The main performance boosts will be validation after the checkpoint and block validation after it's fully synced and running.

With the current small average block sizes I'd see the faster validation being primarily useful for mining as a normal node doesn't need to rush to validate a new block so long as it doesn't take 10 minutes.

At bigger block sizes it becomes more important.

Also I don't think we skip validation of scripts that we've already validated in the mempool. That would be an optimization.

[cpacia]

After doing these benchmarks, what's your opinion. Do you think the speed boost is worth moving away from pure go? I think if we go this route by default then we will kill the cross-compiling that we use for the release.

I wonder if it's possible to use a build tag and have it skip the cgo dep by default and people who want to use it can manually compile?

[zquestz]

We could include both libraries and conditionally include them based on the platform. Technically we can keep cross compilation this way.

[nshopik]

As for the sync speed. Since the most recent checkpoint is Nov 15

Is this also landed in first beta? I'm doing inital sync bchd and it feels like it isn't just download blocks till nov15 but actually verify all stuff.

[cpacia]

It is in the beta