can't decode BTC address about bitcoin-etl

[jingli85]

After the bitcoin core update to release 22.0, we started to see all BTC addresses are converted to nonstandard by bitcoin-etl.

From the 22.0 release note https://bitcoincore.org/en/releases/22.0/, we see
The following RPCs: gettxout, getrawtransaction, decoderawtransaction, decodescript, gettransaction, and REST endpoints: /rest/tx, /rest/getutxos, /rest/block deprecated the following fields (which are no longer returned in the responses by default): addresses, reqSigs. The -deprecatedrpc=addresses flag must be passed for these fields to be included in the RPC response. This flag/option will be available only for this major release, after which the deprecation will be removed entirely. Note that these fields are attributes of the scriptPubKey object returned in the RPC response. However, in the response of decodescript these fields are top-level attributes, and included again as attributes of the scriptPubKey object. (#20286)

The root cause of this issue is that core 22.0 changed the raw json structure. Pre 22.0, it was:
"scriptPubKey": {
"asm": "OP_DUP OP_HASH160 b2f7ff4dc4bb0402aec4664a683c4c0650b1f915 OP_EQUALVERIFY OP_CHECKSIG",
"hex": "76a914b2f7ff4dc4bb0402aec4664a683c4c0650b1f91588ac",
"reqSigs": 1,
"type": "pubkeyhash",
"addresses": [
"1HKJNJLdgXh79WPgR325fq4wb6opqV5Nqr"
]
}
With core 22.0, it is
"scriptPubKey": {
"asm": "OP_DUP OP_HASH160 b2f7ff4dc4bb0402aec4664a683c4c0650b1f915 OP_EQUALVERIFY OP_CHECKSIG",
"hex": "76a914b2f7ff4dc4bb0402aec4664a683c4c0650b1f91588ac",
"address": "1HKJNJLdgXh79WPgR325fq4wb6opqV5Nqr",
"type": "pubkeyhash"
}
The array of "addresses" is replaced with "address"

To fix the problem, we modified
1, the line at this https://github.com/blockchain-etl/bitcoin-etl/blob/master/bitcoinetl/mappers/transaction_output_mapper.py#L49, from output.addresses = script_pub_key.get('addresses') to output.addresses = [script_pub_key.get('address')] in transaction_output_mapper.py
2, the set nonstandard address at here

bitcoin-etl/bitcoinetl/service/btc_service.py

Line 158 in 2d0012f

if output.addresses is None or len(output.addresses) == 0:

accordingly.

We are testing the change right now to see if the problem is fixed or not.

[vu-opstech]

I got same the issue! Still waiting for correct data :D

[akhorshidi]

The same issue exists here and the simple workaround for me was to directly send JSON-RPC calls to the web server started by bitcoind and then do some parsing for each type of the sctipts. For example for scriptPubKey of type P2PK we have:

def script_pub_key_to_pub_key_hash(script_pub_key: str, type: str = 'pubkey'):
    """
    Create a public key hash from scriptPubKey hex

    :param script_pub_key: scriptPubKey hex in hexadecimal notation
    :type script_pub_key: str
    :param type: type of the scriptPubKey
    :type type: str
    :return: hash of public key for P2PK address
    :rtype: bytes
    """
    try:
        if type == 'pubkey':
            # Extract Public Key from ScriptPubKey
            pubkey_hex = script_pub_key[2:-2]
            # The number of bytes in the Public Key
            print(f'The length of public key is {int("0x" + script_pub_key_hex[:2], base=16)} bytes.')
            # Convert Public Key hex too binary
            pubkey_bin = binascii.unhexlify(pubkey_hex)
            # Create SHA-256 hash from Public Key binary
            pub_key_hash = hashlib.sha256(pubkey_bin).digest()
            # Compute RIPEMD-160 hash value/digest
            ripemd160 = hashlib.new('ripemd160')
            ripemd160.update(pub_key_hash)
            pub_key_double_hash = ripemd160.digest()

            return pub_key_double_hash

    except Exception as e:
        logging.info("Exception occurred: ", e, exc_info=True)


def pub_key_hash_to_addr(pubkeyhash: bytes, version_prefix: bytes = b'\x00'):
    """
    Create a Base58Check-encoded address from public key hash

    :param pubkeyhash: hash of public key
    :type pubkeyhash: bytes
    :param version_prefix: a version byte added to hash
    :type version_prefix: bytes
    :return: An bitcoin address in Base58Check format
    :rtype: bytes
    """
    try:
        # First add version byte to get a padded hash
        hash_versioned = version_prefix + pubkeyhash
        # Apply the SHA256 hash algorithm twice
        hash_first = hashlib.sha256(hash_versioned).digest()
        hash_second = hashlib.sha256(hash_first).digest()
        # Add the first four bytes as checksum
        hash_checksum = hash_versioned + hash_second[:4]
        # Encode in Base58Check
        base58check_encoded_address = base58.b58encode(hash_checksum)

        return base58check_encoded_address

    except Exception as e:
        logging.info("Exception occurred: ", e, exc_info=True)


if __name__ == "__main__":
    script_pub_key_hex = "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac"
    utxo_hash = script_pub_key_to_pub_key_hash(script_pub_key_hex)
    utxo_addr = pub_key_hash_to_addr(utxo_hash)
    print(utxo_addr)

HTH

[medvedev1088]

This issue should be fixed in the latest version https://github.com/blockchain-etl/bitcoin-etl/pull/57/files

[akhorshidi]

This issue should be fixed in the latest version https://github.com/blockchain-etl/bitcoin-etl/pull/57/files

I'm using the latest version committed here and get the same output yet: 77f1b46

[medvedev1088]

@akhorshidi do you mind adding steps to reproduce? It works correctly for edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60 posted above

[akhorshidi]

410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac

How to reproduce the bug:
export_blocks_and_transactions --provider-uri http://user:pass@bitcoind:8332 --chain bitcoin --start-block 119965 --end-block 119965 --blocks-output output/blocks.json --transactions-output output/transactions.json

Output Log:
{"hash": "d70ae1131d433b655d0faeae1db4efa15bb4138f1e38e60a53073a58ea1dcc34", "size": 135, "virtual_size": 135, "version": 1, "lock_time": 0, "block_number": 119965, "block_hash": "00000000000069cf03c847d7d1d58b44474021b31dc65a8e83fd3a08e60a768c", "block_timestamp": 1303671399, "is_coinbase": true, "index": 0, "inputs": [], "outputs": [{"index": 0, "script_asm": "0400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41 OP_CHECKSIG", "script_hex": "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac", "required_signatures": null, "type": "nonstandard", "addresses": ["nonstandard2214070acbb06e77ad479bb181885c2c4702d758"], "value": 5012000000}], "input_count": 0, "output_count": 1, "input_value": 0, "output_value": 5012000000, "fee": 0}

As you can see, it labels all the addresses of type PubKey as nonstandard, but bitcoin-cli reports the correct public key which then can be used to generate the corresponding UTXO address, same as ones the we'll see in the explorers like this

bitcoin-cli getrawtransaction d70ae1131d433b655d0faeae1db4efa15bb4138f1e38e60a53073a58ea1dcc34 1

"vout": [
    {
      "value": 50.12000000,
      "n": 0,
      "scriptPubKey": {
        "asm": "0400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41 OP_CHECKSIG",
        "hex": "410400d0ade32217e076945e0946ef7bed72d9aea035aa8891e4bf0749ae6e24f8a7d3ea56efafe472ac3943dbed3af7c093729720ac9ab04e8eba09286e3a00fe41ac",
        "type": "pubkey"
      }
    }
  ],

[medvedev1088]

Ah I see, thanks for the steps.

bitcoin-etl doesn't currently parse scripts and only relies on addresses returned by bitcoin-cli. There is a feature request for this #43