square / beancounter

Utility to audit the balance of Hierarchical Deterministic (HD) wallets. Supports multisig + segwit wallets.

Beancounter

Beancounter is a command line utility to audit the balance of Hierarchical Deterministic (HD) wallets at a given point in time (or block height). The tool is designed to scale and work for wallets with a large number of addresses or a large number of transactions, with support ranging from simple watch wallets to more complicated multisig + segwit. If you're curious, here's why we decided to write Beancounter in the first place.

Beancounter currently supports two types of backends to query the blockchain:

1. Electrum public servers. When using these servers, Beancounter behaves in a similar fashion to an Electrum client wallet. The servers are queried for transaction history for specific addresses. Using Electrum servers is easiest but requires trusting public servers to return accurate information. There is also potential privacy exposure.

2. Private Btcd node. Btcd is a Bitcoin full node which implements transaction indexes. Setting up a Btcd node can take some time (the initial sync takes ~7 days) and requires maintaining the node up-to-date. The benefit is however a higher level of guarantee that the transaction history is accurate.

Getting Started

Installing

Requires Go 1.11 or higher.

$ git clone https://github.com/square/beancounter/
$ cd beancounter
$ go build -mod=vendor

Deriving the child pubkey

Let's imagine we want to track the balance of tpubD8L6UhrL8ML9... and the derivation being used is m/1'/1234/change/index.

We need to manually perform the 1234 derivation:

$ ./beancounter keytree 1234
Enter pubkey #1 out of #1:
tpubD8L6UhrL8ML9Ao47k4pmdvUoiA6QUJVzrJ9BXLgU9idRKnvdRFGgjcxmVxojWGvCcjMi6QWCp8uMpCwWdSFRDNJ7utizxLy27sVWXQT4Jz7
Child pubkey #1: tpubDBrCAXucLxvjC9n9nZGGcYS8pk4X1N97YJmUgdDSwG2p36gbSqeRuytHYCHe2dHxLsV2EchX9ePaFdRwp7cNLrSpnr3PsoPLUQqbvLBDWvh

We can then use tpubDBrCAXucLxvj... to compute the balance.

Compute balance of a HD wallet (using Electrum)

$ ./beancounter compute-balance --type multisig --block-height 1438791
Enter pubkey #1 out of #1:
tpubDBrCAXucLxvjC9n9nZGGcYS8pk4X1N97YJmUgdDSwG2p36gbSqeRuytHYCHe2dHxLsV2EchX9ePaFdRwp7cNLrSpnr3PsoPLUQqbvLBDWvh
...
Balance: 267893477

Compute balance of a single address (using Electrum)

$ ./beancounter compute-balance --type single-address --lookahead 1 --block-height 1438791
Enter single address:
mzoeuyGqMudyvKbkNx5dtNBNN59oKEAsPn
...
Balance: 111168038

Compute balance of a HD wallet (using Btcd)

https://github.com/btcsuite/btcd contains information on how to setup a node.

Beancounter requires addrindex=1, txindex=1, and notls=1. If your node is on a remote server, we recommend tunneling the RPC traffic over ssh or some other secure tunnel.

rpcuser=mia
rpcpass=ilovebrownies
rpclisten=127.0.0.1:8334
notls=1

blocksonly=1
addrindex=1
txindex=1

Once the Btcd is up and running, you can do:

$ ./beancounter compute-balance --type multisig --block-height 1438791 --backend btcd --addr localhost:8334 --rpcuser mia --rpcpass ilovebrownies
Enter pubkey #1 out of #1:
tpubDBrCAXucLxvjC9n9nZGGcYS8pk4X1N97YJmUgdDSwG2p36gbSqeRuytHYCHe2dHxLsV2EchX9ePaFdRwp7cNLrSpnr3PsoPLUQqbvLBDWvh
...
Balance: 267893477

Details

Beancounter is implemented in Go. We picked Go because we wanted an easy build and distribution process. We appreciate the ability to cross-compile and distribute static binaries which don't have external dependencies.

We use the following process to determine a deterministic wallet's balance at a given block height:

1. Derive external (receive) and internal (change) addresses.
2. For each address, query the backend for a list of transactions. We keep deriving additional addresses until we find a large number of unused addresses.
3. Prune the transaction list to remove transactions newer than the block height.
4. For each transaction, query the backend for the raw transaction.
5. For each transaction, track whether the output belongs to the wallet and whether it got spent.
6. Iterate over the transactions and compute the final balance.

Contributing

We appreciate any pull request which fixes bugs or adds features!

If you need ideas on how to contribute, we would enjoy a 3rd backend (Bitcoin-core based, processing each block by streaming the entire blockchain) as well as additional wallet types (e.g. multisig non-segwit).

Additional unittests and improvements to comments/docs are also always welcome.

License

Copyright 2018 Square, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

beancounter's Issues

find-block returns incorrect result for very recent and future timestamps

We should fail (instead of returning an incorrect) result if find-block lands on current-height - 11.

Otherwise, invocations such as ./beancounter find-block "2022-01-01 12:00:00 GMT" will return a different result (until we cross 2022).

Build artifacts in CI

At minimum would be great to have binaries built on Travis for a git tag. As a bonus we could auto-update Github Releases as well.

Default server should be localhost for Btcd

Improve address deriver

Address deriver can be improved in many ways:

• don't assume m=1, n=1 for non-multisig addresses. Instead, take an explicit address type parameter.
• don't assume multisig == segwit. It should be two flags.
• support the full range of address types.

how to verify a normal address

how to check simple address 1NarpHgn2moL6Hes1kMPRLfPtgaM8aPxtm

fixture cache inconsistency issue

There's an issue with fixtures which makes it look like our cache is inconsistent. Figure out why this happens. We have a temporary workaround (reschedule things), but we should find the root cause.

Blacklisted nodes should be eventually retried

In the electrum backend, we should retry blacklisted nodes (either using a time decay or when there are no other nodes left). Otherwise, we risk getting (and staying) disconnected from the network.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

remove potential infinite loops

When the backend is processing data and something goes wrong, the request is re-queued (with the hope that another node will successfully handle the work). This creates a potential infinite loop. You can see this happen if you try to process an address with a huge number of transactions using the Electrum backend (e.g. 18cBEMRxXHqzWWCxZNtU91F5sbUNKhL5PX has 12k transactions, too many for Electrum).

When we re-queue the work, we should track how many attempts we have done. If the number of attempts is too high, we should panic().

Implement integration tests

We have a bunch of unittests, but no integration tests.

We should probably think of integration testing at the same time as #43, as they both might require specific architectural changes.

single-address should imply lookahead=1

Currently, you can invoke beancounter with --type single-address but forget to set lookahead. Lookahead will default to 100, performing a ton of useless work.

Should we track dust?

Small outputs cannot be used (because the fee per byte x bytes required to use the output is greater than the output) and are called dust.

Should we track dust and report the total balance in a wallet followed by the amount of dust? This would give a more accurate view of the actual usable balance.

Failed negotiate version

Hey guys,

I'm trying to run the beancounter tool but receive a Failed negotiate version error:

go run ./main.go compute-balance --type multisig --block-height 1438791 --debug

paste public key:
tpubDBrCAXucLxvjC9n9nZGGcYS8pk4X1N97YJmUgdDSwG2p36gbSqeRuytHYCHe2dHxLsV2EchX9ePaFdRwp7cNLrSpnr3PsoPLUQqbvLBDWv

Result is:

2020/10/19 19:27:58 connecting to testnet.hsmiths.com
2020/10/19 19:27:59 172.88.21.24:53012 <- {"id":1,"method":"server.features","params":[]}
2020/10/19 19:27:59 172.88.21.24:53012 -> {"jsonrpc": "2.0", "result": {"hosts": {"testnet.hsmiths.com": {"ssl_port": 53012, "wss_port": 53014}}, "pruning": null, "server_version": "ElectrumX 1.15.0", "protocol_min": "1.4", "protocol_max": "1.4.2", "genesis_hash": "000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943", "hash_function": "sha256", "services": ["ssl://testnet.hsmiths.com:53012", "wss://testnet.hsmiths.com:53014"]}, "id": 1}
2020/10/19 19:27:59 172.88.21.24:53012 <- {"id":2,"method":"server.version","params":["beancounter","1.2"]}
2020/10/19 19:28:00 172.88.21.24:53012 -> {"jsonrpc": "2.0", "error": {"code": 1, "message": "unsupported protocol version: 1.2"}, "id": 2}
2020/10/19 19:28:00 error on recv from 172.88.21.24:53012: server error (1: unsupported protocol version: 1.2)
panic: Failed negotiate version

goroutine 1 [running]:
github.com/square/beancounter/utils.PanicOnError(...)
	/Users/ilia/Sandbox/beancounter/utils/utils.go:13
main.doComputeBalance()
	/Users/ilia/Sandbox/beancounter/main.go:229 +0xbe5
main.main()
	/Users/ilia/Sandbox/beancounter/main.go:71 +0x1ca
exit status 2

Do you plan to support the v1.4 protocol?

Document that we filter blocks which are unconfirmed or too recent

To ensure reproducible results, we filter some blocks:

When computing a balance:
Beancounter only computes balances up-to "current height" - 6 blocks. We discard the last 6 blocks + unconfirmed blocks.

When mapping a timestamp to a block number:
We can only find blocks in the range 6 to current height - 11 (because of the way we take the median).

support saving/resuming

Given that the tool can take a few hours in some cases, we should support saving the state / resuming.

We need to be careful about not using outdated state if the parameters change. We could simply throw the entire state away, or be smarter and keep some partial state forever (e.g. tx data never changes).

implement progress bar

We know which block number we want to reach and we know (roughly) what block numbers the transaction we are fetching belong to. We can thus implement a progress bar.

Handle addresses with large number of transactions

We make a few assumptions which can lead us to miss transactions if there are too many. For example, we assume the electrum servers are not pruning transactions. We assume a single request is enough to get the full address history. Etc.

Instead of silently returning an incorrect balance, we should at least panic() if an address has too many transactions.

Ensure we only connect to up-to-date electrum nodes

It's possible for an electrum server to have stale data. When we connect to a server, we should do a height check.

improve compute-balance output

We should go back to showing something like this:

+---------------+--------------------+-------------------+---------------------+-------+
| TOTAL BALANCE | LAST RECEIVE INDEX | LAST CHANGE INDEX |     REPORT TIME     | BLOCK |
+---------------+--------------------+-------------------+---------------------+-------+
|        123456 |                 28 |                11 | 26 Sep 18 13:38 PDT | 12345 |
+---------------+--------------------+-------------------+---------------------+-------+

a few (minor) off-by-ones

• min confirmations doesn't take into account that current height is a confirmed block.
• lookahead is off-by-one

fixtures assume you haven't changed any arguments

fixtures only work correctly if all the arguments are unchanged. We should store some kind of hash(params) where params contains the xpubs, single-address, m, lookahead, block-height, etc. and panic if there's a mismatch.

Figure out why electrum backend fails to return correct balance in some cases

We have one or more bugs we still need to squash.