blockchain-etl / bitcoin-etl Goto Github PK

@medvedev1088
Hi,
I found that there ara two kinds of output in transactions.json.
One is nonstandard, and the other is pubkeyhash.
I thought the nonstandard output is the op_return output, but i found outputs of many (not all ) coinbase txs also are nonstandard. And address of miner is like“nonstandard3318537dfb3135df9f3d950dbdf8a7ae68dd7c7d”.
Here is an example:
transaction: {
"hash": "4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b", "block_number": 0,
"block_hash": "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", "is_coinbase": true,
"outputs": [{"index": 0, "script_asm": "04678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5f OP_CHECKSIG", "script_hex": "4104678afdb0fe5548271967f1a67130b7105cd6a828e03909a67962e0ea1f61deb649f6bc3f4cef38c4f35504e51ec112de5c384df7ba0b8d578a4c702b6bf11d5fac", "required_signatures": null, "type": "nonstandard", "addresses": ["nonstandard3318537dfb3135df9f3d950dbdf8a7ae68dd7c7d"], "value": 5000000000}], "input_count": 0, "output_count": 1, "input_value": 0, "output_value": 5000000000, "fee": 0}
The address above is 1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa according to [https://www.blockchain.com/btc/tx/4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b]

I'm confused. Perhaps you know why?
Thank you very much!!!!

The scope of this task is:

1. Implement a job for exporting blocks and transactions using export_blocks_and_transactions.py as a basis.
2. Implement tests using test_export_blocks_job.py as a basis.
3. Implement Dockerfile
4. Write documentation in README using Exporting the Blockchain as a basis.

Don't worry about copy-pasting the code from Ethereum ETL for now. Common code will be factored out later.

In the export_all process, seems the enriching process took the most time, how to speed up this process?

Hi,

I noted that the current, fixed dependency versions of requests, dateutil and click clash with dependencies for e.g. pandas.

Would it be possible to adjust the Python dependency definitions in setup.py?

Locally I've successfully tested

'requests ~= 2.20',
'python-dateutil ~= 2.7',
'click ~= 7.0'

The field should contain the index of a transaction within a block.

@medvedev1088 Hi,
I want to get the values of transactions inputs. so I use enrich_transactions.
An error occurred at runtime:
File "..../bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -5, 'message': 'No such mempool transaction. Use -txindex or provide a block hash to enable blockchain transaction queries. Use gettransaction for wallet transactions.'},
'[{"jsonrpc": "2.0", "method": "getrawtransaction", "params": ["899652e48f685736a1b9af0d2db2e64a906d74064b79650c173b85073aac5d4f", 1], "id": "1"},
...
It dosen't seem to be a problem of bitcoin core.
I will appreciate it if you can give me some advice!
Thank you very much!!

Hi there,

Although I succeeded bittcoinetl get_block_range_for_date command, I got following errors and failed when I run bitcoinetl export_all command.

$ bitcoinetl export_all -s 0 -e 573154 -b 100 -p http://bitcoin:[email protected]:8332 -c bitcoin
2019-04-25 12:03:05,537 - export_all [INFO] - Exporting blocks 00000000-00000099 to output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.json
2019-04-25 12:03:05,537 - export_all [INFO] - Exporting transactions from blocks 00000000-00000099 to output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.json
2019-04-25 12:03:05,537 - ProgressLogger [INFO] - Started work. Items to process: 100.
Traceback (most recent call last):
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 68, in _export
    total_items=self.end_block - self.start_block + 1
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 59, in get_blocks
    return self.get_blocks_by_hashes(block_hashes, with_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 67, in get_blocks_by_hashes
    block_detail_response = self.bitcoin_rpc.batch(block_detail_rpc)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
    raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -1, 'message': 'JSON value is not a boolean as expected'}, '[{"jsonrpc": "2.0", "method": "getblock", "params": ["00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048", 2], "id": "1"}]')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rascal/.local/bin/bitcoinetl", line 11, in <module>
    sys.exit(cli())
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rascal/.local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 101, in export_all
    output_dir, provider_uri, max_workers, export_batch_size)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 98, in export_all
    job.run()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 85, in _end
    self.batch_work_executor.shutdown()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
    self.executor.shutdown()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/rascal/.local/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 59, in get_blocks
    return self.get_blocks_by_hashes(block_hashes, with_transactions)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 67, in get_blocks_by_hashes
    block_detail_response = self.bitcoin_rpc.batch(block_detail_rpc)
  File "/home/rascal/.local/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 53, in batch
    raise ValueError('"result" is None in the JSON RPC response {}. Request: {}', resp_item.get('error'), text)
ValueError: ('"result" is None in the JSON RPC response {}. Request: {}', {'code': -1, 'message': 'JSON value is not a boolean as expected'}, '[{"jsonrpc": "2.0", "method": "getblock", "params": ["000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f", 2], "id": "1"}]')

I use Ubuntu 18.04 LTS, Python version is 3.6.8, and I got result by running bitcoin-cli getblockchaininfo

$ bitcoin-cli -rpcuser=bitcoin -rpcpassword=local321 getblockchaininfo
{
  "chain": "main",
  "blocks": 573170,
  "headers": 573170,
  "bestblockhash": "0000000000000000000411ca1744b51606c9f222b5df1ac4a6b566e7bcf6ce57",
  "difficulty": 6353030562983.983,
  "mediantime": 1556191378,
  "verificationprogress": 0.9999995394033131,
  "chainwork": "000000000000000000000000000000000000000005f6153034b35a5d4cb02cbf",
  "pruned": false,
  "softforks": [
    {
      "id": "bip34",
      "version": 2,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    },
    {
      "id": "bip66",
      "version": 3,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    },
    {
      "id": "bip65",
      "version": 4,
      "enforce": {
        "status": true,
        "found": 1000,
        "required": 750,
        "window": 1000
      },
      "reject": {
        "status": true,
        "found": 1000,
        "required": 950,
        "window": 1000
      }
    }
  ],
  "bip9_softforks": [
    {
      "id": "csv",
      "status": "active"
    }
  ]
}

ethreum-etl is working well, by the way...

Thank you,

FileExporter outputs blocks and transactions to a file partitioned by day/hour. It should have the same interface as https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/jobs/exporters/google_pubsub_item_exporter.py.

It will be used for streaming e.g.

bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000 --output file://output_dir/

https://github.com/blockchain-etl/bitcoin-etl/blob/master/bitcoinetl/mappers/transaction_mapper.py#L41

Maybe place a disclaimer because I just found out that the hash was actually representing transaction_id.

Confirm support for Litecoin

how can i get all addresses on bitcoin blockchain i mean every address that comited a transaction on blockchain no matter it has balance or not?
for example get list of all addresses in text file
no balance or transaction or anything else just adresses of all time
thank you

Parse the bitcoin data file to retrieve transaction and block data

Error message:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cloudinary 1.40.0 requires urllib3>=1.26.5, but you have urllib3 1.24.3 which is incompatible.

Steps to reproduce error:

Step 1:
pip install bitcoin-etl[streaming]

Error will get:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cloudinary 1.40.0 requires urllib3>=1.26.5, but you have urllib3 1.24.3 which is incompatible.

Try to solve error:
i use: pip install urllib3 --upgrade,

then another dependency conflict arise

after so many tries to installing various dependecy version,

when finally i run pip install bitcoin-etl[streaming] i get no error, but after that

on running bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 840000

i get error: ModuleNotFoundError: No module named 'urllib3.packages.six.moves'

this all about dependecny conflict,

i dont know much and dont have much experience if there is any mistake writing in post, i am sorry

I see that there are several old issues and some open pull-requests. I would like to know if this project is still being maintained?

environment :
python | bitcoinetl |
3.6 | 1.3.1 |

[root@localhost end_block=00100199]# bitcoinetl export_all --provider-uri http://user:pass@localhost:8332 --start 2010-01-30   --end   2013-12-30 

confront the error:

Traceback (most recent call last):
  File "/usr/local/bin/bitcoinetl", line 11, in <module>
    load_entry_point('bitcoin-etl==1.3.1', 'console_scripts', 'bitcoinetl')()
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib64/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 102, in export_all
    output_dir, provider_uri, max_workers, export_batch_size, enrich)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 44, in export_all
    for batch_start_block, batch_end_block, partition_dir, *args in partitions:
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 60, in get_partitions
    batch_start_block, batch_end_block = btc_service.get_block_range_for_date(start_date)
  File "/usr/local/lib/python3.6/site-packages/bitcoinetl/service/btc_block_range_service.py", line 37, in get_block_range_for_date
    start_datetime = (date + timedelta(hours=start_hour, minutes=0, seconds=0)).replace(tzinfo=timezone.utc)
TypeError: 'tzinfo' is an invalid keyword argument for this function

what should I handle the error ?

Hi, I was wondering if bitcoin-etl will be supporting the BTC taproot soft fork coming in November. The changes are outlined in BIP340, BIP341, and BIP342.

Thanks!

As you have already implemented, getblock JSON PRC has 2 versions.
Old one doesn't support verbose option, you need to call getrawtransaction RPC to get transaction data.

As long as I know, REST API "blocks" with ".json" extension returns transaction data even if the Coins don't support verbose option on their JSON RPC.
https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md#blocks

How about supporting REST API "block"?
I know it is not elegant, but it has an advantage on speed.

To make REST API enable, you need add args "-rest" with your bitcoind.

Hello :)

Command run

I get 401 client error when running the following command :

bitcoinetl export_blocks_and_transactions --start-block 0 --end-block 500 --provider-uri http://crypto:password@localhost:8332 --chain bitcoin --
blocks-output blocks.json --transactions-output transactions.json

Config

• OS: Ubuntu 20.04.1 LTS
• Python version: Python 3.6.12
• Bitcoinetl version: 1.4.0

Full stack trace

2020-12-14 11:05:58,417 - ProgressLogger [INFO] - Started work. Items to process: 501.
Traceback (most recent call last):
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 58, in get_blocks
    block_hashes = self.get_block_hashes(block_number_batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 88, in get_block_hashes
    block_hashes_response = self.bitcoin_rpc.batch(block_hash_rpc)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 45, in batch
    timeout=self.timeout
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/request.py", line 19, in make_post_request
    response.raise_for_status()
  File "/home/crypto/.etl/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://crypto:password@localhost:8332/

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/crypto/.etl/bin/bitcoinetl", line 8, in <module>
    sys.exit(cli())
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/crypto/.etl/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/cli/export_blocks_and_transactions.py", line 67, in export_blocks_and_transactions
    job.run()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 85, in _end
    self.batch_work_executor.shutdown()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
    self.executor.shutdown()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 68, in _export
    total_items=self.end_block - self.start_block + 1
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/crypto/.etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 64, in _fail_safe_execute
    work_handler([item])
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_blocks_job.py", line 72, in _export_batch
    blocks = self.btc_service.get_blocks(block_number_batch, self.export_transactions)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 58, in get_blocks
    block_hashes = self.get_block_hashes(block_number_batch)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 88, in get_block_hashes
    block_hashes_response = self.bitcoin_rpc.batch(block_hash_rpc)
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 45, in batch
    timeout=self.timeout
  File "/home/crypto/.etl/lib/python3.6/site-packages/bitcoinetl/rpc/request.py", line 19, in make_post_request
    response.raise_for_status()
  File "/home/crypto/.etl/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: http://crypto:password@localhost:8332/

Others

I can curl the endpoint without problem. curl localhost:8332 returns:

JSONRPC server handles only POST requests(.etl) 

Below is the current state of the blockchain (returned by bitcoin-cli -getinfo):

{
  "version": 200100,
  "blocks": 661319,
  "headers": 661319,
  "verificationprogress": 0.9999942484540776,
  "timeoffset": -1,
  "connections": 10,
  "proxy": "",
  "difficulty": 18670168558399.59,
  "chain": "main",
  "balance": 0.00000000,
  "keypoolsize": 1000,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "warnings": ""
}

Thanks for your help !

Could we delineate whether Zcash transaction components are connected to the sprout versus sapling pool? Right now, they are all marked as type=shielded with no way to differentiate. Thanks!

Address 15pHcPCSw9DfkJTD8uQY4okzrayNm9LiA6 in transaction 0b1f0fee9ac088422531af0cf18f9b24024f783ff5426a6b683c2d5082fce652 is not correctly parsed in stream

bitcoinetl stream  --start-block 134916 --end-block 134916 -p http://node  -B 1

it is parsed as nonstandard897bc364defc7cb7b0e4382dc605c897c7707f35

but is correctly stated in the google cloud dataset.

How is this done?

Are these issues related?
#55 #43

https://unlockt.me/v/225ff04e7e

I am trying to extract all transactions from bitcoin blockchain using the script below:
bitcoinetl export_all --start 505200 --end 593600 --partition-batch-size 100 --provider-uri http://localhost:8332 --chain bitcoin --output-dir /data1/bitcoinetl/ --max-workers 10 --export-batch-size 100 --enrich True

But at the block number range of 505200 - 505299, the following error consistently occurs:
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/cli/export_all.py", line 102, in export_all
output_dir, provider_uri, max_workers, export_batch_size, enrich)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/export_all.py", line 117, in export_all
job.run()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 30, in run
self._end()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 103, in _end
self.batch_work_executor.shutdown()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 68, in shutdown
self.executor.shutdown()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 39, in shutdown
self._check_completed_futures()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/jobs/base_job.py", line 28, in run
self._export()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 54, in _export
self.batch_work_executor.execute(self.transactions_iterable, self._enrich_transactions)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 48, in execute
self.executor.submit(self._fail_safe_execute, work_handler, batch)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 31, in submit
self._check_completed_futures()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/blockchainetl/executors/batch_work_executor.py", line 64, in _fail_safe_execute
work_handler([item])
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 63, in _enrich_transactions
input_transactions_map = self._get_input_transactions_as_map(transaction_input_batch)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/jobs/enrich_transactions.py", line 82, in _get_input_transactions_as_map
transactions = self.btc_service.get_transactions_by_hashes(transaction_hashes)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 96, in get_transactions_by_hashes
raw_transactions = self._get_raw_transactions_by_hashes_batched(hashes)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 121, in _get_raw_transactions_by_hashes_batched
result.extend(self._get_raw_transactions_by_hashes(batch))
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/service/btc_service.py", line 134, in _get_raw_transactions_by_hashes
transaction_detail_response = self.bitcoin_rpc.batch(transaction_detail_rpc)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 48, in batch
response = self._decode_rpc_response(raw_response)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/site-packages/bitcoinetl/rpc/bitcoin_rpc.py", line 71, in _decode_rpc_response
return json.loads(response_text, parse_float=decimal.Decimal)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/init.py", line 367, in loads
return cls(**kw).decode(s)
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/data/python/anaconda3/envs/blockchain-etl/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 134037348 (char 134037347)

Does anyone know how to fix it?

For transactions containing Sapling shielded inputs or outputs, a single fake input or output is added to the transaction representing the net migration of funds into or out of the Sapling pool.

However, this fake input or output is not included in the fee calculation, causing fees to appear massively inflated in some cases. Looking at the transaction implementation, the fee should include the fake input or output:

My guess is that this is an ordering problem: the fee is being cached somewhere before the fake input or output is added. I can't see where that is happening though.

As an example, BigQuery says that transaction 13d78b6036aa5be6646c4fa96e3a779eb03f82fdc226e7cf82a9197d28d03a6b has a fee of 294206950000 zatoshis, when it was really 0.0001 ZEC (10,000 zatoshis) according to a blockchain explorer. However, it returns inputs.value = 147103480000 and outputs.value = -147103470000, and adding these together gives the real fee.

Hello,
anyone can help to fix this:

x
Out[178]: <bitcoinetl.domain.block.BtcBlock at 0x7f71d337b780>
x.transactions[1].outputs[1].addresses
Out[179]: ['nonstandardafad2456e5fd6732199d77546c9526255fa66b77']

But bitcoin-cli returns everything correct

root@b:~# bitcoin-cli getrawtransaction edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60 1
{
  "txid": "edd9212daaeb9ac51ce6f5a8b7ff096c74d4aada71615beb9cd1e0858d5ebe60",
  "hash": "5361e2dd2475f86433fd7cf409ca96884a9f53bfcd20bda16cdcdeaf23d74ca0",
  "version": 1,
  "size": 382,
  "vsize": 192,
  "weight": 766,
  "locktime": 0,
  "vin": [
    {
      "txid": "e960fdabe8e301846d64b924766ffc4aba94a8fb056c096e7e6c9b631f5e344e",
      "vout": 0,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c01",
        "30440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c81501",
        "52210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae"
      ],
      "sequence": 4294967295
    }
  ],
  "vout": [
    {
      "value": 0.00913448,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 1683db0ca249360942d1a9dabe5d0cced0c5a903 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac",
        "address": "1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr",
        "type": "pubkeyhash"
      }
    },
    {
      "value": 0.00682552,
      "n": 1,
      "scriptPubKey": {
        "asm": "0 701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "hex": "0020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d",
        "address": "bc1qwqdg6squsna38e46795at95yu9atm8azzmyvckulcc7kytlcckxswvvzej",
        "type": "witness_v0_scripthash"
      }
    }
  ],
  "hex": "010000000001014e345e1f639b6c7e6e096c05fba894ba4afc6f7624b9646d8401e3e8abfd60e90000000000ffffffff0228f00d00000000001976a9141683db0ca249360942d1a9dabe5d0cced0c5a90388ac386a0a0000000000220020701a8d401c84fb13e6baf169d59684e17abd9fa216c8cc5b9fc63d622ff8c58d040047304402200efe981c24a248b5ee2957a06d592220999a33eed783f93384f06b059e5613fd0220085b445a1638ded282aad9dd49120df2d954a4d307266967fbcad2e54f57328c014730440220452e0f0bfbba23e93cc265b04fd9c96cf88d9d1ccc49370444b46ef35f6a199802202507471dc999f2bd8bf4ea16d63a71c02153398002b685365228cf9e99d9c815016952210279d1f38c1c80d47cb00ddbbe2915a60d5706e1ef66056a169150f083b288eb952102cb7d02b654f8616bfc5ab017b7a3ec9092e466381af0f552b7efcd8d920453672103c96d495bfdd5ba4145e3e046fee45e84a8a48ad05bd8dbb395c011a32cf9f88053ae00000000",
  "blockhash": "000000000000000000086a2cff995563e22ed9ed046dc8b56690130893d6175c",
  "confirmations": 1,
  "time": 1637151616,
  "blocktime": 1637151616
}

so correct address for this transaction should be: 1343piJzM5DB4N3gDE7VF17nnRD5FVj4pr

Edit: This is actually expected, and we must run the enrich command to have transaction inputs added. Sorry.

https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/service/graph_operations.py

Should partition by blocks or by date, same as Ethereum ETL's export_all https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/cli/export_all.py.

Level 2 Quotes
https://act.webullfintech.com/totalView/introduce.html?subscriptionGroupUuid=nasdaq-totalview&is_subscribed=false&isForeign=true&effectType=null&expireTime=0&trial=false&month_item_id=subscription.us.nasdaq.lv2.1m&p=$2.99&sourcePage=Stocks&isForeign=true&theme=2&color=1&hl=en&android_sdk_int=31&canary-version=&_v=1&sp=1&statusBarHeightV2=22&wbFontSetting=small&wbFontUnit=20&wbFontSize=14

This is the error:
$ pip install bitcoin-etl
Collecting bitcoin-etl
Could not find a version that satisfies the requirement bitcoin-etl (from versions: )
No matching distribution found for bitcoin-etl

I have a BigQuery where I order all transactions. At the moment, that order is block_number, hash. I'd much rather have transactions in their original block order, but there doesn't seem to be a transaction_index column in the data so it's not possible to recreate the true transaction order. Sorting by hash is a work around for now, but if possible I think adding transaction index would be very helpful as all users can now agree on the same ordering. Thank you!

The documentation mentions that the datasets are updated every 24 hours.

Till when is the daily updation going to be supported for the bitcoin and ehtereum datasets?

When I export the transactions to json in the following manner,

bitcoinetl export_blocks_and_transactions --start-block 500000 --end-block 500001 \
--provider-uri [REMOVED] --chain bitcoin \
--batch-size 100 --transactions-output transactions500000to500001.json

the transactions that I get all have a negative (or 0) mining fee. Here's a specific example.

{
   "hash":"d89197f886482cdb9d0828d69b19b8cb3d3831cded628250bd0506fac687313c",
   "size":226,
   "virtual_size":226,
   "version":1,
   "lock_time":0,
   "block_number":500000,
   "block_hash":"00000000000000000024fb37364cbf81fd49cc2d51c09c75c35433c3a1945d04",
   "block_timestamp":1513622125,
   "is_coinbase":false,
   "inputs":[
      {
         "index":0,
         "spent_transaction_hash":"f2926dd6fdc48342b0fc791ff9d7d7db77420e7e55eaf6d9d8d3a058b6eb113f",
         "spent_output_index":38,
         "script_asm":"3045022100c3b2715a5173fb4aa5a21cceb323e83e74b24fee5d616b700241bd09d1736709022055ec015e30c5f020bda62e6bbb693a4768df9cc8152c93e4c1fd8696165b8a20[ALL] 0211cc09ad5f8a10e8ed6ea995a67bb1142ce5d85a2655290e37f04596745220aa",
         "script_hex":"483045022100c3b2715a5173fb4aa5a21cceb323e83e74b24fee5d616b700241bd09d1736709022055ec015e30c5f020bda62e6bbb693a4768df9cc8152c93e4c1fd8696165b8a2001210211cc09ad5f8a10e8ed6ea995a67bb1142ce5d85a2655290e37f04596745220aa",
         "sequence":4294967295,
         "required_signatures":null,
         "type":null,
         "addresses":[

         ],
         "value":null
      }
   ],
   "outputs":[
      {
         "index":0,
         "script_asm":"OP_DUP OP_HASH160 3e9c503b43723ed5deb1fa33d86b93d4c186b347 OP_EQUALVERIFY OP_CHECKSIG",
         "script_hex":"76a9143e9c503b43723ed5deb1fa33d86b93d4c186b34788ac",
         "required_signatures":1,
         "type":"pubkeyhash",
         "addresses":[
            "16i49jsjaVdAKZVXebziBxa3RjwB1T1Qh5"
         ],
         "value":1149000
      },
      {
         "index":1,
         "script_asm":"OP_DUP OP_HASH160 3287defaad1ab8ec836a355692ac1717dcac9ca3 OP_EQUALVERIFY OP_CHECKSIG",
         "script_hex":"76a9143287defaad1ab8ec836a355692ac1717dcac9ca388ac",
         "required_signatures":1,
         "type":"pubkeyhash",
         "addresses":[
            "15cBZ5MDCJmGLmbLPubyBJBESv1WP2jyDx"
         ],
         "value":2197064
      }
   ],
   "input_count":1,
   "output_count":2,
   "input_value":0,
   "output_value":3346064,
   "fee":-3346064
}

If I'm streaming the fees are of the correct value.

A transaction with more than 13k inputs in this block causes the exception mentioned in the title https://explorer.zcha.in/blocks/618827.

Possible solutions:

• Remove unused fields from JSON payload.
• Use Protocol Buffers instead of JSON.
• Compress data.

Another example: https://explorer.zcha.in/blocks/619473

When I run the export_blocks_and_transactions command in CLI with any start and end block range as such:

bitcoinetl export_blocks_and_transactions --start-block 683753 --end-block 683754 \
provider-uri  quicknode_uri \
blocks-output blocks.json --transactions-output transactions.json

I'm noticing that input_value for the transactions.json file are always returned as 0 and within the inputs array, the value object is also being returned as null. Similarly, fee is also being returned as equal to the output_value. Here is an example transaction returned from the command above:

{
  "hash": "8874ab3b4a000e638f7560987720ae412cf1afc770b31708e51a57d077d37591",
  "size": 195,
  "virtual_size": 113,
  "version": 1,
  "lock_time": 0,
  "block_number": 683753,
  "block_hash": "0000000000000000000a7967a02fa70131044c41ebefc2092d1b6b758e6aa265",
  "block_timestamp": 1621108418,
  "is_coinbase": false,
  "index": 39,
  "inputs": [
    {
      "index": 0,
      "spent_transaction_hash": "77d531fc73bf1dda5b23a3f5f361973d405486f62425d4f7fdf4096cfe11838b",
      "spent_output_index": 38,
      "script_asm": "",
      "script_hex": "",
      "sequence": 4294967295,
      "required_signatures": null,
      "type": null,
      "addresses": [],
      "value": null
    }
  ],
  "outputs": [
    {
      "index": 0,
      "script_asm": "OP_DUP OP_HASH160 ecc64ca8b679689131c54dd78c762408b0923bb4 OP_EQUALVERIFY OP_CHECKSIG",
      "script_hex": "76a914ecc64ca8b679689131c54dd78c762408b0923bb488ac",
      "required_signatures": 1,
      "type": "pubkeyhash",
      "addresses": [
        "1Nax2Mqa99aZdKD5sJyZQzB6FY63GZudf8"
      ],
      "value": 1843616
    }
  ],
  "input_count": 1,
  "output_count": 1,
  "input_value": 0,
  "output_value": 1843616,
  "fee": -1843616
}

You can see that the input_count is 1 yet input_value is 0 while the value within the input array is null. According to the block explorer the input value should be 0.01859552 BTC:

https://www.blockchain.com/btc/tx/8874ab3b4a000e638f7560987720ae412cf1afc770b31708e51a57d077d37591

Similarly fee is incorrect as it is always set equal to -output_value, which also doesn't match the block explorer.

The other possiblity is that something is wrong with the node I'm pulling from (using a node run by quicknode). Anyone else run into similar issues?

GCSExporter outputs blocks with transactions to a file (one per block) and uploads it to a GCS bucket. It should have the same interface as https://github.com/blockchain-etl/bitcoin-etl/blob/master/blockchainetl/jobs/exporters/google_pubsub_item_exporter.py.

It will be used for streaming e.g.

bitcoinetl stream -p http://user:pass@localhost:8332 --start-block 500000 --output gs://bucket_name/folder

The bucket can then be configured to trigger events to Cloud Functions or Pub/Sub.

The JSON files are not supported to upload as an attachment. I captured part of a file. All the input arrays in the files downloaded are empty. What's the problem?

On my environment, the field "type" & "value" at transaction_input are always null.
like,
"inputs": [{"index": 0, "spent_transaction_hash": "100a776847c80201ca6efa5605b806ec770c105af95972eb522e45af385b3892", "spent_output_index": 0, "script_asm": "3045022100b61b8132060e810387518dbd697f731eb1680debf3882ed04821a5d54ced2bca022017a5356096f7db93d9b989114e248476d37325aab7ebc2a8f5800a2e9e6a6618[ALL] 047bade8836f9ccd1923851351e70683606d86a22b6c78676ea3eae343aff5c8c3964e39c9edcf11807949bd325c9475b14de3dcd91b354a7e9d64a0e9b3e10f6b", "script_hex": "483045022100b61b8132060e810387518dbd697f731eb1680debf3882ed04821a5d54ced2bca022017a5356096f7db93d9b989114e248476d37325aab7ebc2a8f5800a2e9e6a66180141047bade8836f9ccd1923851351e70683606d86a22b6c78676ea3eae343aff5c8c3964e39c9edcf11807949bd325c9475b14de3dcd91b354a7e9d64a0e9b3e10f6b", "type": null, "sequence": 4294967295, "value": null},

I know, Getting these values requires another RPC "getrawtransaction" and bitcoin-cli with "getrawtransaction" returns these values on my environment.
Has "getrawtransaction" not been implemented?

I'm using bitcoinetl, version 1.1.0, installed from pip.

See ethereum-etl for an example https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/streaming/eth_streamer_adapter.py#L96.

environment:
macOS | python3.8 | pip 21.0.1

Running installation:
pip install bitcoin-etl

Error:

ERROR: Could not find a version that satisfies the requirement bitcoin-etl 
ERROR: No matching distribution found for bitcoin-etl

Any suggestions? Thanks!!!

From https://github.com/bitcoin/bitcoin the bitcoin core has been making changes in 22.0,
rpc: remove deprecated addresses and reqSigs from rpc outputs via this commit bitcoin/bitcoin@8721638

This change in the core 22.0 require the bitcoin-etl to make the corresponding changes.