blockchain-etl / ethereum-etl Goto Github PK

Python scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Google BigQuery https://goo.gl/oY5BCQ

Home Page: https://t.me/BlockchainETL

License: MIT License

Python 99.21% Shell 0.59% Dockerfile 0.03% HTML 0.16%

ethereum csv export erc20 erc20-tokens etl transaction sql aws gcp

ethereum-etl's Introduction

Ethereum ETL

Ethereum ETL lets you convert blockchain data into convenient formats like CSVs and relational databases.

Do you just want to query Ethereum data right away? Use the public dataset in BigQuery.

Full documentation available here.

Quickstart

Install Ethereum ETL:

pip3 install ethereum-etl

Export blocks and transactions (Schema, Reference):

> ethereumetl export_blocks_and_transactions --start-block 0 --end-block 500000 \
--blocks-output blocks.csv --transactions-output transactions.csv \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Export ERC20 and ERC721 transfers (Schema, Reference):

> ethereumetl export_token_transfers --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/geth.ipc --output token_transfers.csv

Export traces (Schema, Reference):

> ethereumetl export_traces --start-block 0 --end-block 500000 \
--provider-uri file://$HOME/Library/Ethereum/parity.ipc --output traces.csv

Stream blocks, transactions, logs, token_transfers continually to console (Reference):

> pip3 install ethereum-etl[streaming]
> ethereumetl stream --start-block 500000 -e block,transaction,log,token_transfer --log-file log.txt \
--provider-uri https://mainnet.infura.io/v3/7aef3f0cd1f64408b163814b22cc643c

Find other commands here.

For the latest version, check out the repo and call

> pip3 install -e . 
> python3 ethereumetl.py

Useful Links

Running Tests

> pip3 install -e .[dev,streaming]
> export ETHEREUM_ETL_RUN_SLOW_TESTS=True
> export PROVIDER_URL=<your_porvider_uri>
> pytest -vv

Running Tox Tests

> pip3 install tox
> tox

Running in Docker

Install Docker: https://docs.docker.com/get-docker/

Build a docker image

 > docker build -t ethereum-etl:latest .
 > docker image ls

Run a container out of the image

 > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 0 -e 5499999 -b 100000 -p https://mainnet.infura.io
 > docker run -v $HOME/output:/ethereum-etl/output ethereum-etl:latest export_all -s 2018-01-01 -e 2018-01-01 -p https://mainnet.infura.io

Run streaming to console or Pub/Sub

 > docker build -t ethereum-etl:latest .
 > echo "Stream to console"
 > docker run ethereum-etl:latest stream --start-block 500000 --log-file log.txt
 > echo "Stream to Pub/Sub"
 > docker run -v /path_to_credentials_file/:/ethereum-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/ethereum-etl/credentials_file.json ethereum-etl:latest stream --start-block 500000 --output projects/<your-project>/topics/crypto_ethereum

If running on Apple M1 chip add the --platform linux/x86_64 option to the build and run commands e.g.:

docker build --platform linux/x86_64 -t ethereum-etl:latest .
docker run --platform linux/x86_64 ethereum-etl:latest stream --start-block 500000

Projects using Ethereum ETL

Google - Public BigQuery Ethereum datasets
Nansen - Analytics platform for Ethereum

ethereum-etl's People

Contributors

Stargazers

Watchers

Forkers

mogauvin johndpope jonjet nightwalker89 gitter-badger s2r3dswq mickeystone aroundight linweijun computerality xtuyaowu frank-jing zouyong picf shayanb coinsummer ledata das-research1 sandro-magenio allenday 4knahs suahnkim tokusyu rdimce farooqarahim yaps ayushyachitransh noreentry newalchemylimited jpantunes blockout tboydar briantoliveira obulpathi nosoccomtothom carboneumproject piouslove adolfoeliazat ajguerra101 reptilehaus gzuhlwang chofafa alexchenzl cordo-van-saviour pir8aye 0xrajath nunofernandes-plight ashutoshchauhan13 vitaly-am chainsafe lazzarello chensstudio bitbd83 aadebuger bneiluj johnnysheffield microamp thepgoat internet-com senecatheelder digideskio minsu-daniel-kim thosuperman beamjl amangale komyaka cryptomental jbelke kaiwang36 helper-xie alchemist21 crazydev71 marc0olo thammarongg aashutoshrathi ribeiro458 blockscience huangdehui2013 blockchainana1ytics blockchaindarg direkshan-digital ebarojas mfrishert vtruso propy yazzyyaz ms-building-blocks lanlan3322 hizumi0 emrecolako smasuda zyq20130111 aboussetta macos shawnchong0825 shrubins aitianxiang dbuarque mbeacom csun1992

ethereum-etl's Issues

Cannot get export_all.sh function to run (or any others)

Hi there,
I am running geth, with the --ipcdisable flag because I am storing the blockchain data on an external HD and am therefore also using the --datadir flag. Everything is syncing fine and I have about 90% of the chain downloaded. On my local OSX, I have installed the ethereum-etl package (using Python 3.7) and can see all of the .py programs. However, I am getting the errors below. I am using the file where /geth (under it, /chaindata) and /keystore are stored.

Any suggestion would be greatly appreciated.
Thank you.

TCC-mac:ethereum-etl tuckercahillchambers$ ./export_all.sh -s 0 -e 5800000 -b 100000 -p file://Volumes/HG Wells/ -o output 2018-10-01 10:37:54 Exporting blocks 00000000-00099999 to ./blocks/start_block=00000000/end_block=00099999/blocks_00000000_00099999.csv 2018-10-01 10:37:54 Exporting transactions from blocks 00000000-00099999 to ./transactions/start_block=00000000/end_block=00099999/transactions_00000000_00099999.csv 2018-10-01 10:37:56,441 - ProgressLogger [INFO] - Started work. Items to process: 100000. Traceback (most recent call last): File "/Users/tuckercahillchambers/Desktop/ETL/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute work_handler(batch) File "/Users/tuckercahillchambers/Desktop/ETL/ethereum-etl/ethereumetl/jobs/export_blocks_job.py", line 75, in _export_batch response = self.batch_web3_provider.make_request(json.dumps(blocks_rpc)) File "/Users/tuckercahillchambers/Desktop/ETL/ethereum-etl/ethereumetl/providers/ipc.py", line 57, in make_request with self._lock, self._socket as sock: File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/web3/providers/ipc.py", line 46, in __enter__ self.sock = self._open() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/web3/providers/ipc.py", line 59, in _open return get_ipc_socket(self.ipc_path) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/web3/providers/ipc.py", line 30, in get_ipc_socket sock.connect(ipc_path) FileNotFoundError: [Errno 2] No such file or directory

Error during parsing of results running export_all.sh - Results from nohup.out below

2018-05-16 01:34:54 Exporting blocks 04600000-04699999 to output/start_block=04600000/end_block=04699999/blocks_rpc_output_04600000_04699999.json
2018-05-16 02:21:18 Extracting blocks 04600000-04699999 to output/start_block=04600000/end_block=04699999/blocks_04600000_04699999.csv
2018-05-16 02:21:18 Extracting transactions from blocks 04600000-04699999 to output/start_block=04600000/end_block=04699999/transactions_04600000_04699999.csv
2018-05-16 02:56:10 Exporting ERC20 transfers from blocks 04600000-04699999 to output/start_block=04600000/end_block=04699999/erc20_transfers_04600000_04699999.csv
Traceback (most recent call last):
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/providers/ipc.py", line 175, in make_request
response = self.decode_rpc_response(raw_response)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/providers/base.py", line 60, in decode_rpc_response
return json.loads(to_text(response))
File "/home/user/miniconda3/lib/python3.6/json/init.py", line 354, in loads
return _default_decoder.decode(s)
File "/home/user/miniconda3/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/user/miniconda3/lib/python3.6/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 15302641 (char 15302640)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "export_erc20_transfers.py", line 40, in
events = event_filter.get_all_entries()
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/utils/filters.py", line 109, in get_all_entries
log_entries = self._filter_valid_entries(self.web3.eth.getFilterLogs(self.filter_id))
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/eth.py", line 325, in getFilterLogs
"eth_getFilterLogs", [filter_id],
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/manager.py", line 103, in request_blocking
response = self._make_request(method, params)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/manager.py", line 86, in _make_request
return request_func(method, params)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
return make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
result = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/providers/ipc.py", line 177, in make_request
timeout.sleep(0)
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/utils/threads.py", line 68, in sleep
self.check()
File "/home/user/miniconda3/lib/python3.6/site-packages/web3/utils/threads.py", line 61, in check
raise self
web3.utils.threads.Timeout: 300 seconds
2018-05-16 03:31:53 Exporting blocks 04600000-04699999 took 7019 seconds

token includes not only ERC20/ERC723 but also ERC721(NFT)

Hi,

token epo

Windows - "all pipe instances are busy" error

Hi - thanks a lot for the great scripts. I am trying to run the batch exporter (export_all.sh) in git bash (Windows 7, 64 bit) and get the following error:
File "export_blocks_and_transactions.py", line 57, in
job.run()
File "D:\Git\ethereum-etl\ethereumetl\jobs\base_job.py", line 30, in run
self._end()
File "D:\Git\ethereum-etl\ethereumetl\jobs\export_blocks_job.py", line 80, in _end
super()._end()
File "D:\Git\ethereum-etl\ethereumetl\jobs\batch_export_job.py", line 79, in _end
self.executor.shutdown()
File "D:\Git\ethereum-etl\ethereumetl\executors\fail_safe_executor.py", line 39, in shutdown
self._check_completed_futures()
File "D:\Git\ethereum-etl\ethereumetl\executors\fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "C:\Python35\lib\concurrent\futures_base.py", line 398, in result
return self.__get_result()
File "C:\Python35\lib\concurrent\futures_base.py", line 357, in __get_result
raise self._exception
File "C:\Python35\lib\concurrent\futures\thread.py", line 55, in run
result = self.fn(*self.args, **self.kwargs)
File "D:\Git\ethereum-etl\ethereumetl\jobs\batch_export_job.py", line 73, in _fail_safe_export_batch
self._export_batch(block_number, block_number)
File "D:\Git\ethereum-etl\ethereumetl\jobs\export_blocks_job.py", line 65, in _export_batch
response = self.ipc_wrapper.make_request(json.dumps(blocks_rpc))
File "D:\Git\ethereum-etl\ethereumetl\ipc.py", line 57, in make_request
with self._lock, self._socket as sock:
File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 45, in enter
self.sock = self._open()
File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 58, in _open
return get_ipc_socket(self.ipc_path)
File "C:\Python35\lib\site-packages\web3\providers\ipc.py", line 26, in get_ipc_socket
return NamedPipe(ipc_path)
File "C:\Python35\lib\site-packages\web3\utils\windows.py", line 19, in init
raise IOError(err)
OSError: (231, 'CreateFile', 'All pipe instances are busy.')

Any ideas how to fix? thanks a lot!!!

/ethereum-etl/ethereumetl/jobs/export_contracts_job.py", line 66, in _export_contracts result = response['result']

My another server run into this error:

2018-10-27 02:03:17,921 - ProgressLogger [INFO] - Started work. 2018-10-27 02:03:50,923 - ProgressLogger [INFO] - 5000 items processed. 2018-10-27 02:04:00,381 - ProgressLogger [INFO] - 10000 items processed. Traceback (most recent call last): File "export_contracts.py", line 58, in <module> job.run() File "/root/ethereum-etl/ethereumetl/jobs/base_job.py", line 30, in run self._end() File "/root/ethereum-etl/ethereumetl/jobs/export_contracts_job.py", line 87, in _end self.batch_work_executor.shutdown() File "/root/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 65, in shutdown self.executor.shutdown() File "/root/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/root/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/root/ethereum-etl/ethereumetl/jobs/base_job.py", line 28, in run self._export() File "/root/ethereum-etl/ethereumetl/jobs/export_contracts_job.py", line 56, in _export self.batch_work_executor.execute(self.contract_addresses_iterable, self._export_contracts) File "/root/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/root/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() File "/root/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/root/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute work_handler(batch) File "/root/ethereum-etl/ethereumetl/jobs/export_contracts_job.py", line 66, in _export_contracts result = response['result'] KeyError: 'result' 2018-10-27 02:04:03 An error occurred. Quitting.

Add support for JSON export format

Nested and repeated fields in BigQuery are not supported in CSV https://cloud.google.com/bigquery/docs/loading-data#loading_denormalized_nested_and_repeated_data

Add support for geth traces

The scope of this task:

Design the model for geth traces (trace.py can be used as a basis).
Implement the export and extract jobs.
2.a The export job should use traceBlockByNumber (ethereum/go-ethereum#15516) and output raw JSON responses. It should use callTracer, an example JS API call: debug.traceBlockByNumber('0x250252', {'tracer': 'callTracer'}); the same is available via JSON RPC. The job should use BatchWeb3Provider similar to export_blocks_job.py.
2.b The extract job should take the file with raw JSON responses, parses them, map to our domain entity and output to CSV file.
Write unit tests (real mainnet responses should be used in test cases, so the same tests can be run against geth node).
Add documentation to README (an example can be found here). Include instructions for how to start geth i.e. what options to use (e.g. geth --gcmode archive)

=======================
Notes:

Parity traces are already supported #104.

Filter out ASCII characters not supported by BigQuery

BigQuery fails when trying to load a CSV with ASCII 0 with the following message:

Error: Bad character (ASCII 0) encountered.

We need to check what other characters are not supported in BigQuery and filter them out https://en.wikipedia.org/wiki/ASCII.

https://github.com/medvedev1088/ethereum-etl/blob/master/ethereumetl/jobs/export_tokens_job.py#L64

This should probably a separate python script with filtering logic (not in export_tokens_job.py).

Add Dockerfile

The scope for this task is:

Add Dockerfile (this can be used as a basis). Both ipc and rpc should be supported.
Test it.
Update documentation in README with instructions on how to use it.

Add SQL schemas for receipts, logs, contracts, erc20_tokens to schemas/aws

Why separate transactions from blocks in bq schema?

I'm curious about the decision to separate blocks from transactions in the BigQuery schema. Is it a personal preference of querying style (SQL joins rather than unnesting SQL arrays?) Or is there a performance or technical reason for it?

Python 3.7 "SyntaxError" ?

export_blocks_and_transactions.py --start-block 0 --end-block 500000 --ipc-path \\\\.\\pipe\\geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv
Traceback (most recent call last):
File "C:\Users\Administrator\Desktop\ethereum-etl-master\export_blocks_and_transactions.py", line 4, in
from ethereumetl.ipc import IPCWrapper
File "C:\Users\Administrator\Desktop\ethereum-etl-master\ethereumetl\ipc.py", line 5, in
from web3.providers.ipc import get_default_ipc_path, PersistantSocket
File "C:\Python37\lib\site-packages\web3_init_.py", line 8, in
from web3.main import Web3 # noqa: E402
File "C:\Python37\lib\site-packages\web3\main.py", line 26, in
from web3.providers.eth_tester import (
File "C:\Python37\lib\site-packages\web3\providers_init_.py", line 8, in
from .websocket import WebsocketProvider # noqa: F401
File "C:\Python37\lib\site-packages\web3\providers\websocket.py", line 9, in
import websockets
File "C:\Python37\lib\site-packages\websockets_init_.py", line 3, in
from .client import *
File "C:\Python37\lib\site-packages\websockets\client.py", line 20, in
from .protocol import WebSocketCommonProtocol
File "C:\Python37\lib\site-packages\websockets\protocol.py", line 18, in
from .compatibility import asyncio_ensure_future
File "C:\Python37\lib\site-packages\websockets\compatibility.py", line 15
asyncio_ensure_future = asyncio.async # Python < 3.5
^
SyntaxError: invalid syntax

python --version
Python 3.7.0

Ability to export or query all ever used addresses

can it be used to dump all ever used addresses ?

i know i can do that using "--transactions-output transactions.csv" and then extract tx_from and tx_to then sort and remove duplicates

but how to get the addresses from tx_input that are used in contracts ?
EDIT: now i saw that i can use erc20_transfers.csv erc20_from,erc20_to
and how about the miners ?
EDIT: i can use blocks.csv block_miner

any more addresses hidden in the blockchain ?

Add unit tests to cover error handling

Need test cases to cover error handling in batch_work_executor.py

error in get blocks by date

Hi, I get the following error, wonder if you can advise:
$ python get_block_range_for_date.py --provider-uri=https://mainnet.infura.io --date 2018-01-01
Traceback (most recent call last):
File "get_block_range_for_date.py", line 50, in
start_block, end_block = eth_service.get_block_range_for_date(args.date)
File "D:\git\ethereum-etl-master_230818\ethereumetl\service\eth_service.py", line 35, in get_block_range_for_date
start_datetime = datetime.combine(date, datetime.min.time(), tzinfo=timezone.utc)
TypeError: combine() takes at most 2 arguments (3 given)

Thank you!

Arbitrary column ordering in CSV

CSV files seem to have an arbitrary column ordering which can change from batch to batch.

Add bumpversion

Example usage here https://github.com/ethereum/web3.py

Missing Transfers With export_token_transfers.py

I am trying to fetch all erc-20 tokens that have ever been made for each specific token with export_token_transfers.py. However, I am unable to get all transfers successfully. I am using the following command to export the transfers:

python3 export_token_transfers.py --start-block 0 --end-block 6123389 --provider-uri file:///home/jaap/.local/share/io.parity.ethereum/jsonrpc.ipc --output ../Documents/transfers.csv

However, I am only seeing transfers from block 6110001 and later. Is this a bug somehow with these ETL scripts? Or more likely, does this have something to do with the fact my Parity Node was warp synced?

Export Receipts failed

After running the command to get transaction receipts:

sudo python3 export_receipts_and_logs.py --tx-hashes transactions.csv --ipc-path /mnt/volume/ethereum/geth.ipc --receipts-output receipts.csv

I got the following error:

Traceback (most recent call last):
  File "export_receipts_and_logs.py", line 35, in <module>
    job.run()
  File "/mnt/ethereum-etl/ethereumetl/jobs/base_job.py", line 7, in run
    self._end()
  File "/mnt/ethereum-etl/ethereumetl/jobs/export_receipts_job.py", line 58, in _end
    self.batch_work_executor.shutdown()
  File "/mnt/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 44, in shutdown
    self.executor.shutdown()
  File "/mnt/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 16, in shutdown
    self._check_completed_futures()
  File "/mnt/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 24, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/mnt/ethereum-etl/ethereumetl/jobs/base_job.py", line 5, in run
    self._export()
  File "/mnt/ethereum-etl/ethereumetl/jobs/export_receipts_job.py", line 40, in _export
    self.batch_work_executor.execute(self.tx_hashes_iterable, self._export_receipts)
  File "/mnt/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 22, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/mnt/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 8, in submit
    self._check_completed_futures()
  File "/mnt/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 24, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/mnt/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 41, in _fail_safe_execute
    raise ex
  File "/mnt/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 27, in _fail_safe_execute
    work_handler(batch)
  File "/mnt/ethereum-etl/ethereumetl/jobs/export_receipts_job.py", line 46, in _export_receipts
    receipts = [self.receipt_mapper.json_dict_to_receipt(result) for result in results]
  File "/mnt/ethereum-etl/ethereumetl/jobs/export_receipts_job.py", line 46, in <listcomp>
    receipts = [self.receipt_mapper.json_dict_to_receipt(result) for result in results]
  File "/mnt/ethereum-etl/ethereumetl/utils.py", line 25, in rpc_response_batch_to_results
    raise ValueError('result is null in response {}'.format(response_item))
ValueError: result is null in response {'error': {'code': -32602, 'message': 'invalid argument 0: json: cannot unmarshal hex string without 0x prefix into Go value of type common.Hash'}, 'jsonrpc': '2.0', 'id': 0}

Not really clear what resulted in this error.

Windows - SyntaxError: invalid syntax

C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master>python export_blocks_and_transactions.py --start-block 0 --end-block 500000 \ --ipc-path ~/Library/Ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv
Traceback (most recent call last):
File "export_blocks_and_transactions.py", line 4, in
from ethereumetl.ipc import IPCWrapper
File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in
from web3.providers.ipc import get_default_ipc_path, PersistantSocket
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3_init_.py", line 8, in
from web3.main import Web3 # noqa: E402
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\main.py", line 26, in
from web3.providers.eth_tester import (
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers_init_.py", line 8, in
from .websocket import WebsocketProvider # noqa: F401
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\web3\providers\websocket.py", line 9, in
import websockets
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets_init_.py", line 3, in
from .client import *
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\client.py", line 20, in
from .protocol import WebSocketCommonProtocol
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\protocol.py", line 18, in
from .compatibility import asyncio_ensure_future
File "C:\Users\XPS\AppData\Local\Programs\Python\Python37-32\lib\site-packages\websockets\compatibility.py", line 15
asyncio_ensure_future = asyncio.async # Python < 3.5
^
SyntaxError: invalid syntax

Write tests for export_all command

Add streaming support

https://github.com/orgs/blockchain-etl/projects/1

traceAddress field on Internal Transactions

I want to tell root trace from each transactions using traceAddress field, to make status field before byzantium HF.
The latest version of this ETL tool returns null on traceAddress field for both root trace and end traces.

https://github.com/paritytech/wiki/blob/master/JSONRPC-trace-module.md#traceaddress-field

It would be nice if the tool returns the information to tell root trace from other traces.

recover excutional bit on shell files

I found that on the latest master branch there are no excute bit on shell scripts.
Ex: git ls-files -s export_all.sh
-> 100644

So this script isn’t executable.
Could someone kindly recover execute bits?

Or completely move to python verdion of export_all (I love it) and then remove shell version from master ; )

'code': -32015, 'message': 'VM execution error.', 'data': 'Bad jump destination 2'

Every time I export data from blocks 01000000-01099999, there will be this error:

2018-10-23 15:13:22,817 - ProgressLogger [INFO] - Started work. 2018-10-23 15:13:47,147 - ProgressLogger [INFO] - Finished work. Total items processed: 3053. Took 0:00:24.330398. 2018-10-23 15:13:47,150 - CompositeItemExporter [INFO] - contract items exported: 3053 2018-10-23 15:13:47 Extracting token_address from token_transfers file output/token_transfers/start_block=01000000/end_block=01099999/token_transfers_01000000_01099999.csv 2018-10-23 15:13:47 Exporting tokens from blocks 01000000-01099999 to output/tokens/start_block=01000000/end_block=01099999/tokens_01000000_01099999.csv 2018-10-23 15:13:47,931 - ProgressLogger [INFO] - Started work. Traceback (most recent call last): File "export_tokens.py", line 54, in <module> job.run() File "/home/jar/ethereum-etl/ethereumetl/jobs/base_job.py", line 30, in run self._end() File "/home/jar/ethereum-etl/ethereumetl/jobs/export_tokens_job.py", line 55, in _end self.batch_work_executor.shutdown() File "/home/jar/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 65, in shutdown self.executor.shutdown() File "/home/jar/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown self._check_completed_futures() File "/home/jar/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/home/jar/ethereum-etl/ethereumetl/jobs/base_job.py", line 28, in run self._export() File "/home/jar/ethereum-etl/ethereumetl/jobs/export_tokens_job.py", line 43, in _export self.batch_work_executor.execute(self.token_addresses_iterable, self._export_tokens) File "/home/jar/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 48, in execute self.executor.submit(self._fail_safe_execute, work_handler, batch) File "/home/jar/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 31, in submit self._check_completed_futures() File "/home/jar/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures future.result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/home/jar/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute work_handler(batch) File "/home/jar/ethereum-etl/ethereumetl/jobs/export_tokens_job.py", line 47, in _export_tokens self._export_token(token_address) File "/home/jar/ethereum-etl/ethereumetl/jobs/export_tokens_job.py", line 50, in _export_token token = self.token_service.get_token(token_address) File "/home/jar/ethereum-etl/ethereumetl/service/eth_token_service.py", line 42, in get_token total_supply = self._call_contract_function(contract.functions.totalSupply()) File "/home/jar/ethereum-etl/ethereumetl/service/eth_token_service.py", line 60, in _call_contract_function default_value=None) File "/home/jar/ethereum-etl/ethereumetl/service/eth_token_service.py", line 76, in call_contract_function raise ex File "/home/jar/ethereum-etl/ethereumetl/service/eth_token_service.py", line 70, in call_contract_function result = func.call() File "/home/jar/.local/lib/python3.6/site-packages/web3/contract.py", line 1120, in call **self.kwargs File "/home/jar/.local/lib/python3.6/site-packages/web3/contract.py", line 1369, in call_contract_function return_data = web3.eth.call(call_transaction, block_identifier=block_id) File "/home/jar/.local/lib/python3.6/site-packages/eth_utils/functional.py", line 22, in inner return callback(fn(*args, **kwargs)) File "/home/jar/.local/lib/python3.6/site-packages/web3/eth.py", line 294, in call [transaction, block_identifier], File "/home/jar/.local/lib/python3.6/site-packages/web3/manager.py", line 112, in request_blocking raise ValueError(response["error"]) ValueError: {'code': -32015, 'message': 'VM execution error.', 'data': 'Bad jump destination 2'} 2018-10-23 15:13:54 An error occurred. Quitting.

Add 4byte function signature hashes to contracts.csv (for contract interface detection)

This nice tool allows disassembling contract bytecode https://github.com/tintinweb/ethereum-dasm. Using basicblocks method there https://github.com/tintinweb/ethereum-dasm/blob/master/ethereum_dasm/evmdasm.py allows getting the blocks of the contract, including the initialization block where all functions 4byte hashes should be listed:

An example output:

#> python evmdasm.py -f 0x6080604052600436106100ba576000357c0100000000000000000000000000000000000000000000000000000000900463ffffffff16806316243356146100bf57806338af3eed146100ea5780636e15266a14610141578063834ee4171461016c57806386d1a69f146101975780638da5cb5b146101ae5780639b7faaf0146102055780639e1a4d1914610234578063a4e2d6341461025f578063f2fde38b1461028e578063f83d08ba146102d1578063fa2a899714610300575b600080fd5b3480156100cb57600080fd5b506100d461032f565b6040518082815260200191505060405180910390f35b3480156100f657600080fd5b506100ff610335565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b34801561014d57600080fd5b5061015661035b565b6040518082815260200191505060405180910390f35b34801561017857600080fd5b50610181610361565b6040518082815260200191505060405180910390f35b3480156101a357600080fd5b506101ac610367565b005b3480156101ba57600080fd5b506101c36105e6565b604051808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200191505060405180910390f35b34801561021157600080fd5b5061021a61060b565b604051808215151515815260200191505060405180910390f35b34801561024057600080fd5b5061024961061c565b6040518082815260200191505060405180910390f35b34801561026b57600080fd5b5061027461071b565b604051808215151515815260200191505060405180910390f35b34801561029a57600080fd5b506102cf600480360381019080803573ffffffffffffffffffffffffffffffffffffffff16906020019092919050505061072e565b005b3480156102dd57600080fd5b506102e6610883565b604051808215151515815260200191505060405180910390f35b34801561030c57600080fd5b50610315610954565b604051808215151515815260200191505060405180910390f35b60045481565b600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b60055481565b60035481565b60008060009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161415156103c457600080fd5b600260149054906101000a900460ff1615156103df57600080fd5b600260159054906101000a900460ff161515156103fb57600080fd5b61040361060b565b151561040e57600080fd5b61041661061c565b9050600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1663a9059cbb600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff16836040518363ffffffff167c0100000000000000000000000000000000000000000000000000000000028152600401808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff16815260200182815260200192505050602060405180830381600087803b1580156104ff57600080fd5b505af1158015610513573d6000803e3d6000fd5b505050506040513d602081101561052957600080fd5b8101908080519060200190929190505050507f9cf9e3ab58b33f06d81842ea0ad850b6640c6430d6396973312e1715792e7a91600260009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1682604051808373ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff1681526020018281526020019250505060405180910390a16001600260156101000a81548160ff02191690831515021790555050565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1681565b600080429050600454811191505090565b6000600160009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff166370a08231306040518263ffffffff167c0100000000000000000000000000000000000000000000000000000000028152600401808273ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff168152602001915050602060405180830381600087803b1580156106db57600080fd5b505af11580156106ef573d6000803e3d6000fd5b505050506040513d602081101561070557600080fd5b8101908080519060200190929190505050905090565b600260149054906101000a900460ff1681565b6000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff1614151561078957600080fd5b600073ffffffffffffffffffffffffffffffffffffffff168173ffffffffffffffffffffffffffffffffffffffff16141515156107c557600080fd5b8073ffffffffffffffffffffffffffffffffffffffff166000809054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff167f8be0079c531659141344cd1fd0a4f28419497f9722a3daafe3b4186f6b6457e060405160405180910390a3806000806101000a81548173ffffffffffffffffffffffffffffffffffffffff021916908373ffffffffffffffffffffffffffffffffffffffff16021790555050565b60008060009054906101000a900473ffffffffffffffffffffffffffffffffffffffff1673ffffffffffffffffffffffffffffffffffffffff163373ffffffffffffffffffffffffffffffffffffffff161415156108e057600080fd5b600260149054906101000a900460ff161515156108fc57600080fd5b600061090661061c565b11151561091257600080fd5b4260038190555061093060055460035461096790919063ffffffff16565b6004819055506001600260146101000a81548160ff02191690831515021790555090565b600260159054906101000a900460ff1681565b600080828401905083811015151561097b57fe5b80915050929150505600a165627a7a7230582069642cdfbf8b49cb944b12328add2c81c3557600ed88f94b907c0d547a6251db0029

Inst addr  hex   mnemonic        operand                              xrefs                          description
------------------------------------------------------------------------------------------------------------------------------------------------------
   0 [  0 0x000] PUSH1           0x80                                                                # Place 1 byte item on stack.
   1 [  2 0x002] PUSH1           0x40                                                                # Place 1 byte item on stack.
   2 [  4 0x004] MSTORE                                                                              # Save word to memory.
   3 [  5 0x005] PUSH1           0x04                                                                # Place 1 byte item on stack.
   4 [  7 0x007] CALLDATASIZE                                                                        # Get size of input data in current environment.
   5 [  8 0x008] LT                                                                                  # Lesser-than comparison
   6 [  9 0x009] PUSH2           0x00ba                                                              # Place 2-byte item on stack.
   7 [ 12 0x00c] JUMPI           @0xba                                                               # Conditionally alter the program counter.

   8 [ 13 0x00d] PUSH1           0x00                                                                # Place 1 byte item on stack.
   9 [ 15 0x00f] CALLDATALOAD                                                                        # Get input data of current environment.
  10 [ 16 0x010] PUSH29          0x0100000000000000000000000000000000000000000000000000000000                                # Place 29-byte item on stack.
  11 [ 46 0x02e] SWAP1                                                                               # Exchange 1st and 2nd stack items.
  12 [ 47 0x02f] DIV                                                                                 # Integer division operation.
  13 [ 48 0x030] PUSH4           0xffffffff                                                          # Place 4-byte item on stack.
  14 [ 53 0x035] AND                                                                                 # Bitwise AND operation.
  15 [ 54 0x036] DUP1                                                                                # Duplicate 1st stack item.
  16 [ 55 0x037] PUSH4           0x16243356                                                          # Place 4-byte item on stack.
  17 [ 60 0x03c] EQ                                                                                  # Equality  comparison
  18 [ 61 0x03d] PUSH2           0x00bf                                                              # Place 2-byte item on stack.
  19 [ 64 0x040] JUMPI           @0xbf                                                               # Conditionally alter the program counter.

  20 [ 65 0x041] DUP1                                                                                # Duplicate 1st stack item.
  21 [ 66 0x042] PUSH4           0x38af3eed  ('function beneficiary()')                                # Place 4-byte item on stack.
  22 [ 71 0x047] EQ                                                                                  # Equality  comparison
  23 [ 72 0x048] PUSH2           0x00ea                                                              # Place 2-byte item on stack.
  24 [ 75 0x04b] JUMPI           @0xea                                                               # Conditionally alter the program counter.

  25 [ 76 0x04c] DUP1                                                                                # Duplicate 1st stack item.
  26 [ 77 0x04d] PUSH4           0x6e15266a                                                          # Place 4-byte item on stack.
  27 [ 82 0x052] EQ                                                                                  # Equality  comparison
  28 [ 83 0x053] PUSH2           0x0141                                                              # Place 2-byte item on stack.
  29 [ 86 0x056] JUMPI           @0x141                                                              # Conditionally alter the program counter.

  30 [ 87 0x057] DUP1                                                                                # Duplicate 1st stack item.
  31 [ 88 0x058] PUSH4           0x834ee417                                                          # Place 4-byte item on stack.
  32 [ 93 0x05d] EQ                                                                                  # Equality  comparison
  33 [ 94 0x05e] PUSH2           0x016c                                                              # Place 2-byte item on stack.
  34 [ 97 0x061] JUMPI           @0x16c                                                              # Conditionally alter the program counter.

  35 [ 98 0x062] DUP1                                                                                # Duplicate 1st stack item.
  36 [ 99 0x063] PUSH4           0x86d1a69f  ('function release()')                                  # Place 4-byte item on stack.
  37 [104 0x068] EQ                                                                                  # Equality  comparison
  38 [105 0x069] PUSH2           0x0197                                                              # Place 2-byte item on stack.
  39 [108 0x06c] JUMPI           @0x197                                                              # Conditionally alter the program counter.

  40 [109 0x06d] DUP1                                                                                # Duplicate 1st stack item.
  41 [110 0x06e] PUSH4           0x8da5cb5b  (*ambiguous* 'function ideal_warn_timed(uint256,uint128)')                                # Place 4-byte item on stack.
  42 [115 0x073] EQ                                                                                  # Equality  comparison
  43 [116 0x074] PUSH2           0x01ae                                                              # Place 2-byte item on stack.
  44 [119 0x077] JUMPI           @0x1ae                                                              # Conditionally alter the program counter.

  45 [120 0x078] DUP1                                                                                # Duplicate 1st stack item.
  46 [121 0x079] PUSH4           0x9b7faaf0  ('function lockOver()')                                 # Place 4-byte item on stack.
  47 [126 0x07e] EQ                                                                                  # Equality  comparison
  48 [127 0x07f] PUSH2           0x0205                                                              # Place 2-byte item on stack.
  49 [130 0x082] JUMPI           @0x205                                                              # Conditionally alter the program counter.

  50 [131 0x083] DUP1                                                                                # Duplicate 1st stack item.
  51 [132 0x084] PUSH4           0x9e1a4d19  ('function tokenBalance()')                                # Place 4-byte item on stack.
  52 [137 0x089] EQ                                                                                  # Equality  comparison
  53 [138 0x08a] PUSH2           0x0234                                                              # Place 2-byte item on stack.
  54 [141 0x08d] JUMPI           @0x234                                                              # Conditionally alter the program counter.

  55 [142 0x08e] DUP1                                                                                # Duplicate 1st stack item.
  56 [143 0x08f] PUSH4           0xa4e2d634  ('function isLocked()')                                 # Place 4-byte item on stack.
  57 [148 0x094] EQ                                                                                  # Equality  comparison
  58 [149 0x095] PUSH2           0x025f                                                              # Place 2-byte item on stack.
  59 [152 0x098] JUMPI           @0x25f                                                              # Conditionally alter the program counter.

  60 [153 0x099] DUP1                                                                                # Duplicate 1st stack item.
  61 [154 0x09a] PUSH4           0xf2fde38b  ('function transferOwnership(address)')                                # Place 4-byte item on stack.
  62 [159 0x09f] EQ                                                                                  # Equality  comparison
  63 [160 0x0a0] PUSH2           0x028e                                                              # Place 2-byte item on stack.
  64 [163 0x0a3] JUMPI           @0x28e                                                              # Conditionally alter the program counter.

  65 [164 0x0a4] DUP1                                                                                # Duplicate 1st stack item.
  66 [165 0x0a5] PUSH4           0xf83d08ba  ('function lock()')                                     # Place 4-byte item on stack.
  67 [170 0x0aa] EQ                                                                                  # Equality  comparison
  68 [171 0x0ab] PUSH2           0x02d1                                                              # Place 2-byte item on stack.
  69 [174 0x0ae] JUMPI           @0x2d1                                                              # Conditionally alter the program counter.

  70 [175 0x0af] DUP1                                                                                # Duplicate 1st stack item.
  71 [176 0x0b0] PUSH4           0xfa2a8997                                                          # Place 4-byte item on stack.
  72 [181 0x0b5] EQ                                                                                  # Equality  comparison
  73 [182 0x0b6] PUSH2           0x0300                                                              # Place 2-byte item on stack.
  74 [185 0x0b9] JUMPI           @0x300                                                              # Conditionally alter the program counter.

:loc_0xba

We can assume that all PUSH4 opcodes push 4byte function hashes (although not all of them are for functions), collect all of these and add to contracts.csv

Further, we can use these 4byte signatures to identify different contract interfaces such as ERC20, ERC721, ERC777 etc. For example an ERC20 contract must have:

0x18160ddd - totalSupply()
0x70a08231 - balanceOf(address)
0xa9059cbb - transfer(address,uint256)

ERC721, for example, must also have ownerOf(uint256) method.

This will help separate out transfers in erc20_transfers.csv that come from non-ERC20 contracts (right now this file includes all Transfer events even coming from contracts that are not tokens).

This can also be combined with data exported from www.4byte.directory to see Solidity function names.

Rewrite export_all.sh in python

For cross platform compatibility.

ethtoke 0.01a3 has requirement web~=4.0, byt you'll have web3 4.3.0 wich is incompatible

Traceback (most recent call last):
File "export_blocks_and_transactions.py", line 4, in
from ethereumetl.ipc import IPCWrapper
File "C:\Users\XPS\Documents\Python-Ethereum\ethereum-etl-master\ethereumetl\ipc.py", line 5, in
from web3.providers.ipc import get_default_ipc_path, PersistantSocket
ModuleNotFoundError: No module named 'web3'

Add optional parameter "timezone" to get_block_range_for_date.py

It would be nice if we can pass timezone parameter to get_block_range_for_date.py as an option because I usually use JST(+9) in analyzing rather than UTC.

Add dependency on ethereum-dasm to dependencies.txt

Previously was not possible due to this issue tintinweb/ethereum-dasm#5

TypeError: Address must be provided as a string

I am running into the following traceback when running the command below. As far as I can tell, I followed the proper format as per the documentation. The entire command works fine without the --tokens argument, but in that case I would be extracting every erc-20 token transfer ever made, which is going to take a very very long time :)

jaap@Ubuntu-VM:~/ethereum-etl$ python3 export_token_transfers.py --start-block 0 --end-block 6139406 --provider-uri file:///home/jaap/.ethereum/geth.ipc --output ../Documents/transfers.csv --tokens 0xb056c38f6b7dc4064367403e26424cd2c60655e1
2018-08-13 12:38:39,960 - ProgressLogger [INFO] - Started work. Items to process: 6139407.
Traceback (most recent call last):
  File "export_token_transfers.py", line 60, in <module>
    job.run()
  File "/home/jaap/ethereum-etl/ethereumetl/jobs/base_job.py", line 30, in run
    self._end()
  File "/home/jaap/ethereum-etl/ethereumetl/jobs/export_token_transfers_job.py", line 88, in _end
    self.batch_work_executor.shutdown()
  File "/home/jaap/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 65, in shutdown
    self.executor.shutdown()
  File "/home/jaap/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown
    self._check_completed_futures()
  File "/home/jaap/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/jaap/ethereum-etl/ethereumetl/jobs/base_job.py", line 28, in run
    self._export()
  File "/home/jaap/ethereum-etl/ethereumetl/jobs/export_token_transfers_job.py", line 62, in _export
    total_items=self.end_block - self.start_block + 1
  File "/home/jaap/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 48, in execute
    self.executor.submit(self._fail_safe_execute, work_handler, batch)
  File "/home/jaap/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 31, in submit
    self._check_completed_futures()
  File "/home/jaap/ethereum-etl/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
    future.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/jaap/ethereum-etl/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
    work_handler(batch)
  File "/home/jaap/ethereum-etl/ethereumetl/jobs/export_token_transfers_job.py", line 77, in _export_batch
    event_filter = self.web3.eth.filter(filter_params)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/eth.py", line 332, in filter
    [filter_params],
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/manager.py", line 107, in request_blocking
    response = self._make_request(method, params)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/manager.py", line 90, in _make_request
    return request_func(method, params)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
    response = make_request(method, formatted_params)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 47, in apply_formatters
    formatted_params = formatter(params)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/eth_utils/functional.py", line 22, in inner
    return callback(fn(*args, **kwargs))
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/formatters.py", line 41, in apply_formatter_at_index
    yield formatter(item)
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/rpc_abi.py", line 69, in apply_abi_formatters_to_dict
    [data[field] for field in fields],
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/abi.py", line 442, in map_abi_data
    return pipe(data, *pipeline)
  File "cytoolz/functoolz.pyx", line 589, in cytoolz.functoolz.pipe
  File "cytoolz/functoolz.pyx", line 565, in cytoolz.functoolz.c_pipe
  File "cytoolz/functoolz.pyx", line 236, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/abi.py", line 474, in data_tree_map
    return recursive_map(map_to_typed_data, data_tree)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/decorators.py", line 35, in wrapped
    wrapped_val = to_wrap(*args)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/formatters.py", line 115, in recursive_map
    items_mapped = map_collection(recurse, data)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/formatters.py", line 102, in map_collection
    return datatype(map(func, collection))
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/formatters.py", line 114, in recurse
    return recursive_map(func, item)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/decorators.py", line 35, in wrapped
    wrapped_val = to_wrap(*args)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/formatters.py", line 116, in recursive_map
    return func(items_mapped)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/abi.py", line 471, in map_to_typed_data
    return ABITypedData(func(*elements))
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/normalizers.py", line 45, in wrapper
    modified = to_wrap(abi_type, data)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/normalizers.py", line 124, in abi_address_to_hex
    validate_address(data)
  File "/home/jaap/.local/lib/python3.6/site-packages/web3/utils/validation.py", line 152, in validate_address
    raise TypeError('Address {} must be provided as a string'.format(value))
TypeError: Address ['0xb056c38f6b7dc4064367403e26424cd2c60655e1'] must be provided as a string

No support for HTTPProvider

would be great to connect by JSON RPC.

Data_dir

I just want to use an external data storage (my nas server), but unfortunately I do not understand how to add a data dir folder.

As far as I understand, there are no keys to specifying a specific directory.

Missing contracts created in internal transactions

I have tried to find the address of all verified smart contracts from etherscan.io on google big query ( table bigquery-public-data.ethereum_blockchain.contracts ) and it seems that all contracts that have been created by an internal transaction are missing from the google big query data base. Is that indeed the case?

Here is my query:

#standardSQL
SELECT
  COUNT(verifieds.address)
FROM
  `bocktest-224205.etherscan.contractsVerified` AS verifieds
WHERE
  LOWER(verifieds.address) NOT IN (
  SELECT
    LOWER(contracts.address)
  FROM
    `bigquery-public-data.ethereum_blockchain.contracts` AS contracts)
LIMIT
  10

1406 contracts out of a total of 48839 verified smart contracts are missing...

Here are three examples :

[https://etherscan.io/address/0x381c23667cdb85514b8bc8ebe7406fec9d0c8f3c]
[https://etherscan.io/address/0xada83270510b6284e27c757096d660f0fcf6a90b]
[https://etherscan.io/address/0xa27d94001a2a58395b1bb34ea2f842912a906163]

Try this to see that these internal contracts are indeed missing:

#standardSQL
SELECT
  *
FROM
  `bigquery-public-data.ethereum_blockchain.contracts`
WHERE
  LOWER(address) = '0x381c23667cdb85514b8bc8ebe7406fec9d0c8f3c'
LIMIT
  10

--pieter

Add ethereum-etl to pip

Need the ability to install ethereum-etl via pip

https://packaging.python.org/tutorials/installing-packages/

Write tests for cli commands

http://click.palletsprojects.com/en/7.x/testing/

Here is an example: 3d88dba

Permission denied.

When I ran "ethereumetl export_token_transfers --start-block 5000000 --end-block 6773000 --provider-uri file:/home/jar/jsonrpc.ipc --output /home/jar/token/token_transfers.csv", I got this error:

`jar@jar-B450M-DS3H:~/token$ ethereumetl export_token_transfers --start-block 5000000 --end-block 6773000 --provider-uri file:/home/jar/jsonrpc.ipc --output /home/jar/token/token_transfers.csv
2018-11-26 15:07:07,444 - ProgressLogger [INFO] - Started work. Items to process: 1773001.
Traceback (most recent call last):
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/batch_work_executor.py", line 53, in _fail_safe_execute
work_handler(batch)
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 77, in _export_batch
event_filter = self.web3.eth.filter(filter_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/eth.py", line 332, in filter
[filter_params],
File "/home/jar/.local/lib/python3.6/site-packages/web3/manager.py", line 109, in request_blocking
response = self._make_request(method, params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/manager.py", line 92, in _make_request
return request_func(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
return make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
result = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 206, in make_request
with self._lock, self._socket as sock:
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 46, in enter
self.sock = self._open()
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 59, in _open
return get_ipc_socket(self.ipc_path)
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 30, in get_ipc_socket
sock.connect(ipc_path)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/jar/.local/bin/ethereumetl", line 11, in
sys.exit(cli())
File "/home/jar/.local/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/home/jar/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/home/jar/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/jar/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jar/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/cli/export_token_transfers.py", line 56, in export_token_transfers
job.run()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/base_job.py", line 30, in run
self._end()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 88, in _end
self.batch_work_executor.shutdown()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/batch_work_executor.py", line 65, in shutdown
self.executor.shutdown()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/fail_safe_executor.py", line 39, in shutdown
self._check_completed_futures()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/base_job.py", line 28, in run
self._export()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 62, in _export
total_items=self.end_block - self.start_block + 1
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/batch_work_executor.py", line 48, in execute
self.executor.submit(self._fail_safe_execute, work_handler, batch)
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/fail_safe_executor.py", line 31, in submit
self._check_completed_futures()
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/fail_safe_executor.py", line 47, in _check_completed_futures
future.result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
return self.__get_result()
File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/executors/batch_work_executor.py", line 61, in _fail_safe_execute
work_handler([item])
File "/home/jar/.local/lib/python3.6/site-packages/ethereumetl/jobs/export_token_transfers_job.py", line 77, in _export_batch
event_filter = self.web3.eth.filter(filter_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/eth.py", line 332, in filter
[filter_params],
File "/home/jar/.local/lib/python3.6/site-packages/web3/manager.py", line 109, in request_blocking
response = self._make_request(method, params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/manager.py", line 92, in _make_request
return request_func(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/gas_price_strategy.py", line 18, in middleware
return make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/attrdict.py", line 18, in middleware
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/normalize_errors.py", line 9, in middleware
result = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 50, in apply_formatters
response = make_request(method, params)
File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.call
File "/home/jar/.local/lib/python3.6/site-packages/web3/middleware/formatting.py", line 48, in apply_formatters
response = make_request(method, formatted_params)
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 206, in make_request
with self._lock, self._socket as sock:
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 46, in enter
self.sock = self._open()
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 59, in _open
return get_ipc_socket(self.ipc_path)
File "/home/jar/.local/lib/python3.6/site-packages/web3/providers/ipc.py", line 30, in get_ipc_socket
sock.connect(ipc_path)
PermissionError: [Errno 13] Permission denied`

Invalid rows being written in transaction_*.csv files

Earlier I tried running export_all.sh and am receiving rows that have too many columns. My guess is that the files aren't being properly handled, resulting in erroneous rows being written.

An example I'm seeing of an invalid row:

0x39a53f3f5d5c973134eb9eaecfed1a9d629ca5050bcd34e3ef249a242e539ac0,14310,0x18ed5bb58d11bb6215c0fa05601328f005227d0ed58dead0a006fed1d7812185,500xad9e48cd56d7dd148ac56124db4f247d32d9b3c71d8b36c759b2388cd80e1c3c,14336,0x3199bc051a2c83c911e7a9b78435aba005a25ad1bd3221f2289ee0ca016de379,500487,12,0xf8b483dba2c3b7176a3da549ad41a48bb3121069,0x0a5d9a6fd4688469c78478805ac190a2c6366778,230
08406140000000000,90000,50000000000,0x

It looks like it should be:

0x39a53f3f5d5c973134eb9eaecfed1a9d629ca5050bcd34e3ef249a242e539ac0,14310,0x18ed5bb58d11bb6215c0fa05601328f005227d0ed58dead0a006fed1d7812185,50

0xad9e48cd56d7dd148ac56124db4f247d32d9b3c71d8b36c759b2388cd80e1c3c,14336,0x3199bc051a2c83c911e7a9b78435aba005a25ad1bd3221f2289ee0ca016de379,500487,12,0xf8b483dba2c3b7176a3da549ad41a48bb3121069,0x0a5d9a6fd4688469c78478805ac190a2c6366778,230
08406140000000000,90000,50000000000,0x

It seems that transaction 0x39a53f3f5d5c973134eb9eaecfed1a9d629ca5050bcd34e3ef249a242e539ac0 got cut off and the next row started writing too early.

Add support for internal transactions

Parity provides APIs for retrieving internal transactions. trace_filter API allows retrieving traces for a range of blocks. This method is already supported by web3.py parity module.

The scope of this task is to

Design the model (the field names from this PR can be used as a basis: #72).
Implement the export job (the job implementation will mostly resemble the flow in ExportTokenTransfersJob.
Write unit tests (an example can be found here)
Add documentation to README (an example can be found here)
(5. Add instructions for how to start parity/geth i.e. what options to use).

=======================

Remarks:

There will be another task to add the BigQuery schema here
Geth also provides support for tracing transactions. There will be another task for implementing internal transactions exporter job for Geth. ethereum/go-ethereum#15516, https://www.linkedin.com/pulse/geth-18-iceberg-michael-hunt-jr-

Add GraphQL exporters

Should be done when support is added to geth https://twitter.com/nicksdjohnson/status/1051451048236384257

Implement CLI with Click package

Create Command Line Interface using the Click package.

The CLI should have the following commands (the link references the corresponding command implemented with argparse):

ethereumetl export blocks: https://github.com/blockchain-etl/ethereum-etl#export_blocks_and_transactionspy
ethereumetl export receipts: https://github.com/blockchain-etl/ethereum-etl#export_receipts_and_logspy
ethereumetl export token_transfers: https://github.com/blockchain-etl/ethereum-etl#export_token_transferspy
ethereumetl export tokens: https://github.com/blockchain-etl/ethereum-etl#export_tokenspy
ethereumetl export contracts: https://github.com/blockchain-etl/ethereum-etl#export_contractspy
ethereumetl extract token_transfers: https://github.com/blockchain-etl/ethereum-etl#extract_token_transferspy
ethereumetl get block_range: https://github.com/blockchain-etl/ethereum-etl#get_block_range_for_datepy
ethereumetl get keccak_hash: https://github.com/blockchain-etl/ethereum-etl#get_keccak_hashpy
ethereumetl export all: https://github.com/blockchain-etl/ethereum-etl/blob/develop/export_all.py
ethereumetl export traces: https://github.com/blockchain-etl/ethereum-etl#export_tracespy

Additional requirements:

Click configs should be in ethereumetl.cli package.
In scope of this task also writing documentation in README for all the commands.

======================
Remarks:

Sample Click configurations can be found here https://github.com/analyseether/ether_sql/blob/master/ether_sql/cli/__init__.py

Add support for eth_getTransactionReceipt command

I would like to get a value of "gasUsed" in each transactions at the same time as Blocks and Transactions data exporting.
I think adding support of JSON RPC "eth_getTransactionReceipt" is the best way to get this information, but is it possible to implement to this ETL tool?

Ran export_all command in Windows but only generated erc20_transfer csv

Im running on windows and used the following command:
./export_all.sh -s 3000000 -e 4000000 -b 100000 -i '\.\pipe\geth.ipc' -o output

following error occured:

2018-05-17 23:00:08 Exporting blocks 03000000-03099999 to output/start_block=03000000/end_block=03099999/blocks_rpc_output_03000000_03099999.json
Traceback (most recent call last):
File "exchange_with_ipc.py", line 20, in
response = socket_exchange(args.ipc_path, ''.join(line_batch), args.ipc_timeout)
File "C:\ethereum-etl\ethereumetl\socket_utils.py", line 10, in socket_exchange
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
AttributeError: module 'socket' has no attribute 'AF_UNIX'
Traceback (most recent call last):
File "gen_blocks_rpc.py", line 32, in
output_file.write(json.dumps(data) + '\n')
OSError: [Errno 22] Invalid argument
2018-05-17 23:00:08 Extracting blocks 03000000-03099999 to output/start_block=03000000/end_block=03099999/blocks_03000000_03099999.csv
2018-05-17 23:00:08 Extracting transactions from blocks 03000000-03099999 to output/start_block=03000000/end_block=03099999/transactions_03000000_03099999.csv
2018-05-17 23:00:08 Exporting ERC20 transfers from blocks 03000000-03099999 to output/start_block=03000000/end_block=03099999/erc20_transfers_03000000_03099999.csv
2018-05-17 23:05:18 Exporting blocks 03000000-03099999 took 310 seconds

Error running exchange_with_ipc.py after generating blocks_rpc.json

After generating blocks_rpc.json, the following commmand exchange_with_ipc.py resulted in error.

python exchange_with_ipc.py --ipc-path=~/.local/share/io.parity.ethereum/jsonrpc.ipc --input=blocks_rpc.json --output=blocks_rpc_output.json
Traceback (most recent call last):
File "exchange_with_ipc.py", line 20, in
response = socket_exchange(args.ipc_path, ''.join(line_batch), args.ipc_timeout)
File "/home/user/ethereum-etl/ethereumetl/socket_utils.py", line 13, in socket_exchange
sock.connect(socket_path)
FileNotFoundError: [Errno 2] No such file or directory

Support for continuously polling for new blocks

is it possible to make --end-block latest ? so the script keeps working in real time as soon as a new block comes ?

Python 3.5 is not supported

Executing
python export_blocks_and_transactions.py --start-block 0 --end-block 500000 --ipc-path ~/.ethereum/geth.ipc --blocks-output blocks.csv --transactions-output transactions.csv
unfortunately results in an error
File "export_blocks_and_transactions.py", line 5, in
from ethereumetl.jobs.export_blocks_job import ExportBlocksJob
File "/home/vn/ethereum-etl/ethereumetl/jobs/export_blocks_job.py", line 7, in
from ethereumetl.mappers.block_mapper import EthBlockMapper
File "/home/vn/ethereum-etl/ethereumetl/mappers/block_mapper.py", line 1, in
from ethereumetl.domain.block import EthBlock
File "/home/vn/ethereum-etl/ethereumetl/domain/block.py", line 8
self.number: int = None
^

Add support for Python 3.5 and 3.7

The scope of this task is:

Test that all commands work in Python 3.5 and 3.7.
Change the code where necessary (e.g. f'' string formatting is not supported in Python 3.5)
Write tox config for Python 3.5, 3.6, 3.7 (example tox usage can be found here https://github.com/ethereum/web3.py)
Update README with instructions on how to run tox tests.
Travis CI configuration and testing that it works in Travis CI server.

Notes:

All dependencies in this project support python 3.5 and 3.7. https://pypi.org/project/click/, https://github.com/ethereum/web3.py

Write files in compressed format.

It is very useful if this etl tool could output csv files in compressed format.
Because the size of output files are huge and analyzing tool like pandas can read gzip file directly.

Date based hive partition is more useful than Block No. based in analyzing.

Recently I’ve been analyzing Ethereum block chain data and found that the Block No. based partions didn’t benefit me.
I think the Block No. based partition is useful only when a query exactly fits the start_block and end_block and it didn’t occur in my situation, but querying by dates was occurred so often.
So I would like suggest that export_all.sh and related schemes should be refactored by Date based partition using “get_block_range_for_date.py”