Install EOS ETL:
pip install eos-etlExport blocks, transactions and actions (Schema, Reference):
> eosetl export_blocks --start-block 1 --end-block 500000 \
--provider-uri http://api.main.alohaeos.com \
--blocks-output blocks.json --transactions-output transactions.json --actions-output actions.jsonStream blockchain data continually to console:
> pip install eos-etl[streaming]
> eosetl stream -p http://api.main.alohaeos.com --start-block 500000For the latest version, check out the repo and call
> pip install -e .[streaming]
> python eosetl.py-
Install python 3.6.0+ https://www.python.org/downloads/
-
Install EOS node or get access to EOS node maintained by someone else (because running your own node is not so easy). Some docs:
- https://developers.eos.io/eosio-nodeos/docs/
- https://eosnode.tools/
- https://github.com/CryptoLions/EOS-MainNet
- Make sure it downloaded the blocks that you need by executing in the terminal:
curl --request POST \
--url https://localhost:8080/v1/chain/get_info \
--header 'accept: application/json'You can export blocks below last_irreversible_block_num, there is no need to wait until the full sync
-
Install EOS ETL:
> pip install eos-etl -
Export blocks, transactions and actions:
> eosetl export_all --start 1 --end 499999 \ --provider-uri http://api.main.alohaeos.comIn case
eosetlcommand is not available in PATH, usepython -m eosetlinstead.The result will be in the
outputsubdirectory, partitioned in Hive style:output/blocks/start_block=00000000/end_block=00000099/blocks_00000000_00000099.csv output/blocks/start_block=00000100/end_block=00000199/blocks_00000100_=00000199.csv ... output/transactions/start_block=00000000/end_block=00000099/transactions_00000000_00000099.csv ... output/actions/start_block=00000000/end_block=00000099/actions_00000000_00000099.csv ...
-
Install Docker https://docs.docker.com/install/
-
Build a docker image
> docker build -t eos-etl:latest . > docker image ls
-
Run a container out of the image
> MSYS_NO_PATHCONV=1 docker run -v $HOME/output:/eos-etl/output eos-etl:latest \ export_blocks --max-workers 50 --start-block 30000000 \ --end-block 30000100 --provider-uri http://your_eos_node:node_port \ --blocks-output ./output/blocks.csv --transactions-output ./output/transactions.csv \ --actions-output ./output/actions.csv
-
Run streaming to console or Pub/Sub
> MSYS_NO_PATHCONV=1 docker build -t eos-etl:latest-streaming -f Dockerfile_with_streaming . > echo "Stream to console" > MSYS_NO_PATHCONV=1 docker run eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000 > echo "Stream to Pub/Sub" > MSYS_NO_PATHCONV=1 docker run -v /path_to_credentials_file/:/eos-etl/ --env GOOGLE_APPLICATION_CREDENTIALS=/eos-etl/credentials_file.json eos-etl:latest-streaming stream -p http://api.main.alohaeos.com --start-block 500000 --output projects/your-project/topics/crypto_eos
-
Refer to https://github.com/blockchain-etl/blockchain-etl-streaming for deploying the streaming app to Google Kubernetes Engine.
All the commands accept -h parameter for help, e.g.:
> python eosetl.py export_blocks --help
Usage: eosetl.py export_blocks [OPTIONS]
Export blocks, transactions and actions.
Options:
-s, --start-block INTEGER Start block
-e, --end-block INTEGER End block [required]
-p, --provider-uri TEXT The URI of the remote EOS node
-w, --max-workers INTEGER The maximum number of workers.
--blocks-output TEXT The output file for blocks. If not provided
blocks will not be exported. Use "-" for stdout
--transactions-output TEXT The output file for transactions. If not
provided transactions will not be exported. Use
"-" for stdout
--actions-output TEXT The output file for actions. If not provided
transactions will not be exported. Use "-"
for stdout
--help Show this message and exit.For the --output parameters the supported type is json. The format type is inferred from the output file name.
> python eosetl.py export_blocks --start-block 1 --end-block 500000 \
--provider-uri http://api.main.alohaeos.com \
--blocks-output blocks.json --transactions-output transactions.json --actions-output actions.jsonOmit --blocks-output or --transactions-output or --actions-output options if you want to export only transactions/blocks/actions.
You can tune --max-workers for performance.
> python eosetl.py get_block_range_for_date --provider-uri http://api.main.alohaeos.com --date=2018-06-09> python eosetl.py export_all --provider-uri http://api.main.alohaeos.com --start 2018-06-08 --end 2018-06-09You can tune --export-batch-size, --max-workers for performance.
> python eosetl.py stream --provider-uri http://api.main.alohaeos.com --start-block 500000- This command outputs blocks and transactions to the console by default.
- Use
--outputoption to specify the Google Pub/Sub topic where to publish blockchain data, e.g.projects/your-project/topics/eos_blockchain. - The command saves its state to
last_synced_block.txtfile where the last synced block number is saved periodically. - Specify either
--start-blockor--last-synced-block-fileoption.--last-synced-block-fileshould point to the file where the block number, from which to start streaming the blockchain data, is saved. - Use the
--lagoption to specify how many blocks to lag behind the head of the blockchain. It's the simplest way to handle chain reorganizations - they are less likely the further a block from the head. - You can tune
--period-seconds,--batch-size,--max-workersfor performance.
> pip install -e .[dev]
> echo "The below variables are optional"
> export EOSETL_PROVIDER_URI=http://api.main.alohaeos.com
> pytest -vv> pip install tox
> tox
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent