in3rsha / bitcoin-utxo-dump Goto Github PK

I have a fullnode running from where I copy the chainstate to a new folder to run this script against.

Result of various dumps:

Total UTXOs: 66632964
Total BTC:   18481045.05567317

Total UTXOs: 66632964
Total BTC:   18481045.05567317

Total UTXOs: 66632964
Total BTC:   18481045.05567317

Total UTXOs: 66592544
Total BTC:   18479582.45596703

Total UTXOs: 66529362
Total BTC:   18485795.13070076

Total UTXOs: 66552445
Total BTC:   18488411.92702591

Total UTXOs: 66554994
Total BTC:   18460739.06478096

Total UTXOs: 66530417
Total BTC:   18542498.44333380

Total UTXOs: 66325016
Total BTC:   18472223.00121369

Total UTXOs: 66461702
Total BTC:   18496591.51414967

Total UTXOs: 66472668
Total BTC:   18497154.38937850

The total amount of btc seems to be overstated compared to other sources:
https://statoshi.info/dashboard/db/unspent-transaction-output-set
https://www.blockchain.com/charts/total-bitcoins
https://coinmarketcap.com/currencies/bitcoin/ (circulating supply)

How is that possible?

Trying to run bitcoin-utxo-dump, but getting an error stating

"Couldn't find /.bitcoin/chainstate/"

Ive changed default folder to:

defaultfolder := fmt.Sprintf("I:/Bitcoin/bitcoin/chainstate/", os.Getenv("HOME"))

But getting same error?

Using Windows 10, my primary drive is C:/.

Hi, can I get the utxo set without download the whole transaction history?
Could you please share the set via a google driver ? Many thanks!

Take a look at tx 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098 (block 1). All elements are there except address:

[root@tmp-1-fra bin]# cat utxodump.csv | grep 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098 36012933,0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098,0,5000000000,p2pk, [root@tmp-1-fra bin]#

I sorted utxodump.txt by amount and find the maximum amount is 2147483647 and minimum is negative number.

count,txid,vout,amount,type,address,script
9999824,4f4d5fbae8e4032e04dfe4efb8b7c30d849e2bf4c71eeac6b0f9fbd3bef3fc27,0,2147483647,p2sh,3PSvFA2ooWccdBuJNJAUDunzZBefN344h3,eea833ff2bd9f433a24e9bfbaaf2efed4b8013c7
9999479,36983a7a2bc5b7e197f6c64e40b078a17adffe98e1d9157245f742876d8bfc27,1,2147483647,p2sh,39A8qzNqin9Pb9sBJ1aV8ahTatmtx1m4As,51e9e3a40f69fca7922518272496be878bb99934
9999478,36983a7a2bc5b7e197f6c64e40b078a17adffe98e1d9157245f742876d8bfc27,0,2147483647,p2sh,3FsTxDgQsUj65SxX4611tzPD6HW41QE5CZ,9b8ba5fe8221c8fa22f16c6e2ef52022bca07768
...
21107494,65fb37015b1ae9e3842c34c9da9a6533dd9ac0b88dad303b1e180a8b68ecfd53,0,-238313366,p2sh,3BtaVbiGaZgUVrE5JsK872nFUeq497q8T3,6fe1057bc8d00bca4835ec28675a151bf95a181e
43986610,a93d8cad6698aa2f6c8f8ff4424a9d090befaaf002ef7f60fa74f9256b7852ae,0,-238315865,p2pkh,14mNifRxUzLEt8RanX7BPff9NhVoQm5zUv,294ccf28748923d84f26f54122d5fbe050d23514
20112100,596442355e77fa75d14800750609673e9405f7e96df12bfef5c39bd1a6351150,0,-238415907,p2sh,3NJYSp765xpjo2rFpxzfk6Hc6hiuffGdt2,e21a8d7f42597de1187ddbb3db636a4db9e1389c

hello, i reimplemented this tool in typescript here: https://github.com/Overtorment/bitcoin-utxo-dump-ts
just fyi.
thanks for your work!

Hi, very often, after extraction of the utxo, I get a corrupted db. Is there any way to prevent that? I'm copying around the DB files, making backup & restore, but that's annoying and time consuming after while.
Edit: Here's an example:

2019-12-29T15:21:44Z Verifying last 6 blocks at level 3
2019-12-29T15:21:44Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...ERROR: VerifyDB(): *** coin database inconsistencies found (last 6 blocks, 0 good transactions before that)

2019-12-29T15:21:46Z : Corrupted block database detected.
Please restart with -reindex or -reindex-chainstate to recover.
2019-12-29T15:21:46Z Aborted block database rebuild. Exiting.

Hi and thanks 4 the great work!

I ran this against my full node and it said at the end there where like 2,3 million unspent btc in chainstate.
should this not add up to the 18 million btc created till today..? lost or not they have to be bound in some kind of unspent UTXO.. right ? Or is there something I'm not thinking of or don't know ;-)
it doesn't show multi-sig addresses but since there is a sat amount...
so where are the other 15,7 mil btc

Edit:
On further inspection my file is 1.01 GB .. not 7.xx
then I thought I had a file size limitation so I ran -f amount,address and the file was about 350mb
so i waited and ran against a newer chainstate and the UTXO count and the btc amount went up
now I have 8529954 UTXO's and 2.39 million btc as of mon 18, nov 2019
before I had a UTXO count of 8242975 and 2.29 million btc
therefore I don't think it stops prematurely..

any ideas?
ps: b core is running on newest Mac OS and the .go was also run on Mac OS

I installed this and ran bitcoin-utxo-dump in the last week and repeated today. I noticed that the total BTC is reported out as 19299194.55098371 when it is listed as lower on bitcoinblockhalf.com at 19,254,100. bitcoin.clarkmoody.com/dashboard/ also lists it at about the same as bitcoinblockhalf.

Why?

Hello Sir,

I am wondering if there's anyway to extract the timestamp of UTXO as well.

Thanks in advance!

struggling to add working pubkey point decompression script. I've seen python examples but unable to get the math to workout quite the same in golang at the moment

e.g. https://bitcoin.stackexchange.com/questions/86234/how-to-uncompress-a-public-key

#! /usr/bin/env python3

import binascii

p = 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFEFFFFFC2F

def decompress_pubkey(pk):
x = int.from_bytes(pk[1:33], byteorder='big')
y_sq = (pow(x, 3, p) + 7) % p
y = pow(y_sq, (p + 1) // 4, p)
if y % 2 != pk[0] % 2:
y = p - y
y = y.to_bytes(32, byteorder='big')
return b'\x04' + pk[1:33] + y

print(binascii.hexlify(decompress_pubkey(binascii.unhexlify('0229b3e0919adc41a316aad4f41444d9bf3a9b639550f2aa735676ffff25ba3898'))).decode())
print(binascii.hexlify(decompress_pubkey(binascii.unhexlify('02f15446771c5c585dd25d8d62df5195b77799aa8eac2f2196c54b73ca05f72f27'))).decode())

After execute the export command and save data to csv file. I found that some fields are missing, such as address.

The usage section of the project description reads:

script - The locking script placed on the output (this is just the hash160 public key or hash160 script for a P2PK, P2PKH, or P2SH)

Shouldn't it actually be "this is just the compressed public key for P2PK or hash160 script for a P2PKH and P2SH"? Also for P2WPKH it looks like hash160 prepended with 0014. I have not investigated other types.

I noticed the above when checking out some P2PK transactions in mempool. Can someone explain to me, assuming I'm right about the script output above, why some P2PK have script prefix different than 02 and 03? There are multiple ones with 04 and 05. 04 would be OK if the "script" was just uncompressed public key, but the key length suggests compressed public key and if so, the prefix should be 02 or 03 depending on the parity of Y coordinate, not 04 nor 05.

Example

bitcoin-utxo-dump -db <chainstatepath> -p2pkaddresses -f type,script,address,txid
grep 1Lx9PbLowBtAzMWttPuXhMA5dJzJsvaXMh utxodump.csv
p2pk,05ffffcdeee179409d5b0e3133929d6c1292d57768182e650878304dc6867e7a0b,1Lx9PbLowBtAzMWttPuXhMA5dJzJsvaXMh,d1aa3ca0641ccbd624097847c227448823f0b5d6eb8c0b047335c056886f9084

By the way, why it also returns p2pkh utxo with identical address?
p2pkh,dad84560c59d0cfe43572a7dbf57bfea7a004280,1Lx9PbLowBtAzMWttPuXhMA5dJzJsvaXMh,8309ae00b956bff6e1acf247bdae5d9b7b3ea44c382a32fdb66bb1dd3fa806df

Am I missing something?

bitcoin-cli dumptxoutset utxo.dat

Hi in3rsha - great work on this!

I'm on a Mac and have gone through the installation process up until running $bitcoin-utxo-dump. When doing so I get "command not found". I had to get around "sudo apt install libleveldb-dev" by using "brew install libleveldb" instead (without -dev, since it can't be found via brew), so I'm not sure if this is what's causing the issue but the binary seems to have been created when running "go get github.com/in3rsha/bitcoin-utxo-dump" - it just isn't found for some reason. Any ideas?

Thanks! - Andrew

Hi in3rsha, thank you for the great tool!

I am trying to get a list of UTXOs for Litecoin. Everything is in order except the satoshi values. Do I have to change anything in order to get the correct amount in satoshi?

Hi,

is there a way to identify UTXO that have been spent, and total amount spent for each UTXO?
Sorry, not really an issue, but couldn't find any other forum where I could raise this.

Command used:
bitcoin-utxo-dump -db Backupchainstate/ -f txid,address,amount -o btc_test.txt

example output from btc_test.txt
749fa48af8fab93ee1d8d3f32c4aef147686d394d4725d938cfcd634844c8103,1Ak4TzLZ4pgKYHpr5GontQgoDqwQunAK6c,50000000000
fc7185f9ab7b90161c9b4f6f59fa96e158c671225717e74093a08de5c88e8103,,5000000000
8695751953a8bda3328ab3c14b67c503a4884d323d1fe2887423b7c2a0898203,,5000000000
8769b385bd62d1bf31f6bd8f5868dfcee7798d2f0afb0f535e38295f4aef8203,,5000000000
2c7605bebf15ed82490d2a1fde57bedb9d8de81db38336d2194f1467c8ca8503,,5000000000
1642e0b6243ad8c14f73ddbd8b1d59c1b27fa8edc3527767c0eceb4161118603,,5000000000
2421bcfae5eb1abcaf2224d236bfb72f1a4f730243cd2ab16c0675d822498803,,5000000000
eb016d74bbdc1d1e3debd83d00d352b67e74c051600992a92d0df8d54ff18903,,5000000000
7ab361114598d72857947e721b767fe8e6f0fab577fae547275dea14a7608a03,,5000000000

If any of the txid's with no address is checked in blockexplorer it will show as an initial mining coin transaction.

Is there a switch in this great application that can be used to show the address of these transactions?

Can you publish the latest database file? It will take me 55 hours to download bitcoin core. Maybe there is some repository or site where such abbreviated databases are stored?

Hi @in3rsha,

I've noticed the program is not outputting the correct script for P2PK when the public key is uncompressed.
Example: output of txid 050a20f455b8e98b7a374fd00c810b7c5102ef260b69bea89dd5792e55235fd8 has script 050766aa8462489f7957d5ae8dc499fa580f45f7f143d94eb7e461e2597416005d instead of 040766aa8462489f7957d5ae8dc499fa580f45f7f143d94eb7e461e2597416005db647f4a78c534057fc887667c5c332b287cc5ade652d319cc6942649f20dafdb.

I see 2 issues:
1- The pubkey is only decompressed if address field is selected and flag p2pkaddresses is used.
I think it should also be decompressed if script field is selected.

2- Even if those conditions are met, output["script"] is not updated.

Is this a bug? Can I fix it?

how to get all addresses that had transactions.

Hello. Thanks for the great tool. Is it possible to add support for other similar cryptocurrencies? Interested in Dogecoin.

A feature request: quiet output verbosity level option - do not print any messages to stdout, so the tool can be executed programmatically (by other scripts) without the necessity to handle stdout messages.

Dear sir, I think and hope this will be an easy fix and I'm 99.9% sure this is my bad, but when i try to run "bitcoin-utxo-dump" it gives me a command not found error. I ran the command in various locations including the folder where the executable file is located. I'm aware that there is a similar issue that has been closed already, but that didn't help me fix my issue. If I open the executable it does give me a csv file with the utxo's but i cant use any parameters, i need the block height included. Also a version where height is printed in default will help me fix my issue.

Thanks in advance,
Camiel

I tried it twice with two of my nodes after running your UTXO tool I had to -reindex my nodes.
Is this normal?

Hi, great code. The loop handles the obfuscation key for prefix 14, and utxo data for prefix 67, but I noticed there is a prefix value 66. It is not accounted for in the code. Do you know what this is for?

bitcoin-utxo-dump does not work when bitcoind is running. I am aware this is by design. However, the problem is that the tool does not want to work when bitcoind is running even if I provide a path to a copy of the chainstate folder which is not currently used by any instance of bitcoind and which was copied before, while bitcoind was not running. I suggest replacing the current behavior with a warning and a user prompt if a user wishes to continue, as I recon, detecting if selected folder is an actual bitcoind database or just a copy might be difficult. If this is implemented, I would also appreciate a switch for ignoring this warning, so it can be used programmatically.

I found that use this tool dump out data is less than the data it should be. As the follow I execute the tool and then at the same time get data from rpc.

bitcoin-utxo-dump final result

Total UTXOs: 73333114
Total BTC:   18642162.20453415
Script Types:
 p2wpkh       4044781
 p2wsh        298623
 non-standard 9540
 p2pk         61206
 p2pkh        54670818
 p2sh         13828327
 p2ms         681935

bitcoin-cli getmempoolinfo

{
  "height": 674001,
  "bestblock": "00000000000000000007f8d1652e3bf35c9d2672be4834fef6ebc6f1c407bd8c",
  "transactions": 44795401,
  "txouts": 73360127,
  "bogosize": 5498036952,
  "hash_serialized_2": "7ac46aa0d60b6be1eef2f66e099e8dc8589fd9400792b14697a3361407e15ae6",
  "disk_size": 4935108151,
  "total_amount": 18649807.38006601
}

I was browsing your code and found a bug, so look:

script := xor[offset:]

// P2PKH
if nsize == 0 {

[some code]

scriptType = "p2pkh"
scriptTypeCount["p2pkh"] += 1
}

// P2MS
if len(script) > 0 && script[len(script)-1] == 174 { // if there is a script and if the last opcode is OP_CHECKMULTISIG (174) (0xae)

scriptType = "p2ms"
scriptTypeCount["p2ms"] += 1
}

So any p2pkh address with last byte of hash == 174 will be overwritten as 'p2ms', which is not. It should be switch like.

It doesn't work.

`
Total UTXOs: 5856992

Total BTC: 0.000
`

Can you implement it or just give me a hint how to do it.
Thank you

hey
i need the addresses's file (have all the bitcoin' address).
but i tried many times ,it' s can be ture ,cant install leveldb.
im not pro.
could i get your addresses file
thanks