Comments (42)
allada
commented on June 11, 2024
What kind of hardware are you running on? I suspect the issue might be that the algorithm used to figure out how many pipes to open is a bit too high.
I also do not recommend downloading outside of us-west-2 and instead download to a temp instance in us-west-2 then transfer the data directly to your actual end point. This is because it can take multiple days in some cases to do the download, but there's only ~1-2 days of time before the underlying resource is replaced which can also cause pre-mature pipe failures. By downloading to a us-west-2 instance you should have the files significantly faster.
from bsc-archive-snapshot.
mahirkarim1
commented on June 11, 2024
Thanks for responding. I'm running the recommended Im4gn 4xlarge on us-west-2
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
I setup a m5ad.4xlarge in us-west-2 with 3T gp3, and git clone the code, then run ./build_archive_node.sh under root. Everything is ok when I run it in the afternoon, but later the script stopped at midnight, then I restart ./build_archive_node.sh under root again. But I get the following error:
download: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-022100-022200-transactions-to-block.idx to ../erigon/data/bsc/snapshots/v1-022100-022200-transactions-to-block.idx
download: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-022100-022200-transactions.idx to ../erigon/data/bsc/snapshots/v1-022100-022200-transactions.idx
download: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-022100-022200-transactions.seg to ../erigon/data/bsc/snapshots/v1-022100-022200-transactions.seg
download failed: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-020500-021000-transactions.seg to ../erigon/data/bsc/snapshots/v1-020500-021000-transactions.seg [Errno 28] No space left on device
download failed: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-020000-020500-transactions.seg to ../erigon/data/bsc/snapshots/v1-020000-020500-transactions.seg [Errno 28] No space left on device
download failed: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-021000-021500-transactions.seg to ../erigon/data/bsc/snapshots/v1-021000-021500-transactions.seg [Errno 28] No space left on device
download failed: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/v1-021500-022000-transactions.seg to ../erigon/data/bsc/snapshots/v1-021500-022000-transactions.seg [Errno 28] No space left on device
then I use df -lh, I have 3T gp3, it should not give "No space left on device"
Filesystem Size Used Avail Use% Mounted on
/dev/root 2.9T 6.0G 2.9T 1% /
devtmpfs 31G 0 31G 0% /dev
tmpfs 31G 0 31G 0% /dev/shm
tmpfs 6.2G 948K 6.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 31G 0 31G 0% /sys/fs/cgroup
/dev/loop0 26M 26M 0 100% /snap/amazon-ssm-agent/5656
/dev/nvme0n1p15 105M 5.2M 100M 5% /boot/efi
/dev/loop1 56M 56M 0 100% /snap/core18/2566
/dev/loop2 64M 64M 0 100% /snap/core20/1623
/dev/loop3 68M 68M 0 100% /snap/lxd/22753
/dev/loop4 48M 48M 0 100% /snap/snapd/16778
tmpfs 6.2G 0 6.2G 0% /run/user/1000
/dev/loop5 115M 115M 0 100% /snap/core/13886
/dev/loop6 15M 15M 0 100% /snap/aws-cli/130
/dev/loop7 100M 100M 0 100% /snap/go/9981
tank/erigon_data 18G 128K 18G 1% /erigon/data
tank/erigon_data/bsc 18G 128K 18G 1% /erigon/data/bsc
tank/erigon_data/bsc/parlia 18G 1.4M 18G 1% /erigon/data/bsc/parlia
tank/erigon_data/bsc/snapshots 539G 521G 18G 97% /erigon/data/bsc/snapshots
tank/erigon_data/bsc/chaindata 18G 128K 18G 1% /erigon/data/bsc/chaindata
tank/erigon_data/bsc/nodes 18G 5.7M 18G 1% /erigon/data/bsc/nodes
what I should do to fix this?
from bsc-archive-snapshot.
allada
commented on June 11, 2024
3TB, even with compression is not likely to be able to fit the entire BSC chain. I believe a 3.5TB to 4TB drive would work though.
Keep in mind most of your AWS costs are going to be in EBS storage. This is why ephemeral storage is preferred, it's cheaper and faster.
Lastly, if you stick to EBS, attach many smaller EBS volumes (maybe 10 x 400gig gp3 drives). This will dramatically increase performance (especially download time).
from bsc-archive-snapshot.
allada
commented on June 11, 2024
BTW, using i4i.4xlarge should only take about 2 hours to download and run, i4i.8xlarge takes usually less than an hour.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
I am using i4i.4xlarge, the current df -lh:
/dev/root 7.6G 6.1G 1.6G 80% /
devtmpfs 62G 0 62G 0% /dev
tmpfs 62G 0 62G 0% /dev/shm
tmpfs 13G 924K 13G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 62G 0 62G 0% /sys/fs/cgroup
/dev/nvme0n1p15 105M 5.2M 100M 5% /boot/efi
/dev/loop0 26M 26M 0 100% /snap/amazon-ssm-agent/5656
/dev/loop1 56M 56M 0 100% /snap/core18/2566
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop3 64M 64M 0 100% /snap/core20/1623
/dev/loop4 48M 48M 0 100% /snap/snapd/16778
tmpfs 13G 0 13G 0% /run/user/1000
/dev/loop5 100M 100M 0 100% /snap/go/9981
tank/erigon_data 3.3T 128K 3.3T 1% /erigon/data
tank/erigon_data/bsc 3.3T 128K 3.3T 1% /erigon/data/bsc
tank/erigon_data/bsc/snapshots 3.4T 100G 3.3T 3% /erigon/data/bsc/snapshots
tank/erigon_data/bsc/parlia 3.3T 128K 3.3T 1% /erigon/data/bsc/parlia
tank/erigon_data/bsc/nodes 3.3T 5.7M 3.3T 1% /erigon/data/bsc/nodes
tank/erigon_data/bsc/chaindata 3.3T 128K 3.3T 1% /erigon/data/bsc/chaindata
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Can you post lsblk and zfs list?
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
lsblk:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 25.1M 1 loop /snap/amazon-ssm-agent/5656
loop1 7:1 0 55.6M 1 loop /snap/core18/2566
loop2 7:2 0 67.8M 1 loop /snap/lxd/22753
loop3 7:3 0 63.2M 1 loop /snap/core20/1623
loop4 7:4 0 48M 1 loop /snap/snapd/16778
loop5 7:5 0 99.6M 1 loop /snap/go/9981
zd0 230:0 0 48G 0 disk [SWAP]
nvme1n1 259:0 0 3.4T 0 disk
├─nvme1n1p1 259:5 0 3.4T 0 part
└─nvme1n1p9 259:6 0 8M 0 part
nvme0n1 259:1 0 8G 0 disk
├─nvme0n1p1 259:2 0 7.9G 0 part /
├─nvme0n1p14 259:3 0 4M 0 part
└─nvme0n1p15 259:4 0 106M 0 part /boot/efi
zfs list:
NAME USED AVAIL REFER MOUNTPOINT
tank 170G 3.13T 96K none
tank/erigon_data 170G 3.13T 96K /erigon/data
tank/erigon_data/bsc 170G 3.13T 120K /erigon/data/bsc
tank/erigon_data/bsc/chaindata 96K 3.13T 96K /erigon/data/bsc/chaindata
tank/erigon_data/bsc/nodes 5.57M 3.13T 5.57M /erigon/data/bsc/nodes
tank/erigon_data/bsc/parlia 96K 3.13T 96K /erigon/data/bsc/parlia
tank/erigon_data/bsc/snapshots 170G 3.13T 170G /erigon/data/bsc/snapshots
tank/swap 56K 3.13T 56K -
from bsc-archive-snapshot.
allada
commented on June 11, 2024
It looks like for unknown reasons it is not downloading the tank/erigon_data/bsc/chaindata section. It should happen in parallel with the tank/erigon_data/bsc/snapshots section.
Did you create this i4i.4xlarge then only run the ./build_archive_node.sh?
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
yes, I create i4i.4xlarge, and git clone, and then run as root, then ./build_archive_node.sh, here is the result after done:
- systemctl start erigon-bsc
- add_create_snapshot_script
- set -euxo pipefail
- cat
- chmod 0744 /erigon/create-bsc-snapshot-and-shutdown.sh
- chown root:root /erigon/create-bsc-snapshot-and-shutdown.sh
- echo 'create-bsc-snapshot-and-shutdown /erigon/create-bsc-snapshot-and-shutdown.sh uid=root erigon'
root@ip-172-31-5-127:/bsc-archive-snapshot# df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.6G 6.1G 1.6G 80% /
devtmpfs 62G 0 62G 0% /dev
tmpfs 62G 0 62G 0% /dev/shm
tmpfs 13G 928K 13G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 62G 0 62G 0% /sys/fs/cgroup
/dev/nvme0n1p15 105M 5.2M 100M 5% /boot/efi
/dev/loop0 26M 26M 0 100% /snap/amazon-ssm-agent/5656
/dev/loop1 56M 56M 0 100% /snap/core18/2566
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop3 64M 64M 0 100% /snap/core20/1623
/dev/loop4 48M 48M 0 100% /snap/snapd/16778
/dev/loop5 100M 100M 0 100% /snap/go/9981
tank/erigon_data 2.8T 128K 2.8T 1% /erigon/data
tank/erigon_data/bsc 2.8T 256K 2.8T 1% /erigon/data/bsc
tank/erigon_data/bsc/snapshots 3.3T 599G 2.8T 18% /erigon/data/bsc/snapshots
tank/erigon_data/bsc/parlia 2.8T 128K 2.8T 1% /erigon/data/bsc/parlia
tank/erigon_data/bsc/nodes 2.8T 5.3M 2.8T 1% /erigon/data/bsc/nodes
tank/erigon_data/bsc/chaindata 2.8T 2.2G 2.8T 1% /erigon/data/bsc/chaindata
tmpfs 13G 0 13G 0% /run/user/1000
root@ip-172-31-5-127:/bsc-archive-snapshot# cd /erigon
root@ip-172-31-5-127:/erigon# ls
create-bsc-snapshot-and-shutdown.sh data erigon start_erigon_service.sh
from bsc-archive-snapshot.
allada
commented on June 11, 2024
I see now it shows some data in the chaindata, but not enough.
Can you post the entire output of the script? You can use ./build_archive_node.sh | tee /tmp/output.txt then the data will stream to terminal and then you can copy /tmp/output.txt here.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
root@ip-172-31-5-127:/bsc-archive-snapshot# cat /tmp/output.txt
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Fetched 336 kB in 1s (319 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
60 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists...
Building dependency tree...
Reading state information...
make is already the newest version (4.2.1-1.2).
pv is already the newest version (1.6.6-1).
python3-boto3 is already the newest version (1.9.253-1).
super is already the newest version (3.30.1-1).
unzip is already the newest version (6.0-25ubuntu1.1).
zfsutils-linux is already the newest version (0.8.3-1ubuntu12.14).
clang-12 is already the newest version (1:12.0.0-3ubuntu1~20.04.5).
jq is already the newest version (1.6-1ubuntu0.20.04.1).
0 upgraded, 0 newly installed, 0 to remove and 60 not upgraded.
NAME USED AVAIL REFER MOUNTPOINT
tank 690G 2.63T 96K none
psplit 0.1.0
Takes a file or stdin and splits it by -b bytes and sends them to the
OUTPUT_COMMAND in parallel limited by -p concurrent jobs.
Example:
$ printf '012345678901234567890123' | psplit -b 10 sh -c 'cat > /tmp/psplit_$SEQ; echo >>
/tmp/psplit_$SEQ'
Will result in 3 files:
/tmp/psplit_0 = "0123456789
"
/tmp/psplit_1 = "0123456789
"
/tmp/psplit_2 = "0123
"
USAGE:
psplit [OPTIONS] <OUTPUT_COMMAND>
ARGS:
<OUTPUT_COMMAND> Command to run on each output. An environmental variable "SEQ" will be
set containing the sequence number of the slice. Data will be sent to
stdin of command
OPTIONS:
-b, --bytes <BYTES>
Number of bytes per output file [default: 1073741824]
-h, --help
Print help information
-i, --input-file <INPUT_FILE>
Input file to split. If not set, uses stdin
-p, --parallel-count <PARALLEL_COUNT>
Number of commands allowed to run in parallel [default: 16]
-V, --version
Print version information
pjoin 0.1.0
Takes stdin commands split by new lines and executes them in parallel and
prints stdout in order based on the order they are in stdin.
Example:
$ pjoin <<'EOT'
sh -c 'printf foo; sleep 1'
printf bar
EOT
Will result in:
'foobar' (without quotes)
USAGE:
pjoin [OPTIONS] [OUTPUT_FILE]
ARGS:
<OUTPUT_FILE> Path to write file. Prints to stdout if not set. Using a file can be faster
than stdout
OPTIONS:
-b, --buffer-size <BUFFER_SIZE>
Size in bytes of the stdout buffer for reach command [default: 1073741824]
-h, --help
Print help information
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: 16]
-V, --version
Print version information
Downloads data from s3 and sends it to stdout very fast and on a pinned version.
Many concurrent connections to s3 are opened and different chunks of the file
are downloaded in parallel and stiched together using the `pjoin` utility.
USAGE:
s3pcp [OPTIONS] [S3_PATH]
ARGS:
S3_PATH A path to an s3 object. Format: 's3://{bucket}/{key}'
OPTIONS:
--requester-pays
If the account downloading is requesting to be the payer for
the request.
--region <REGION>
The region the request should be sent to.
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: based on computer resources]
-h, --help
Print help information
erigon [global options] command [command options] [arguments...]
VERSION:
2.28.1-stable-9711af6d
COMMANDS:
init Bootstrap and initialize a new genesis block
import Import a blockchain file
snapshots
help Shows a list of commands or help for one command
GLOBAL OPTIONS:
--datadir value Data directory for the databases (default: "/root/.local/share/erigon") [<invalid Value>]
--ethash.dagdir value Directory to store the ethash mining DAGs (default: "/root/.local/share/erigon-ethash") [<invalid Value>]
--snapshots Default: use snapshots "true" for BSC, Mainnet and Goerli. use snapshots "false" in all other cases
--experimental.lightclient enables experimental CL lightclient.
--txpool.disable experimental external pool and block producer, see ./cmd/txpool/readme.md for more info. Disabling internal txpool and block producer.
--txpool.locals value Comma separated accounts to treat as locals (no flush, priority inclusion)
--txpool.nolocals Disables price exemptions for locally submitted transactions
--txpool.pricelimit value Minimum gas price (fee cap) limit to enforce for acceptance into the pool (default: 1)
--txpool.pricebump value Price bump percentage to replace an already existing transaction (default: 10)
--txpool.accountslots value Minimum number of executable transaction slots guaranteed per account (default: 16)
--txpool.globalslots value Maximum number of executable transaction slots for all accounts (default: 10000)
--txpool.globalbasefeeslots value Maximum number of non-executable transactions where only not enough baseFee (default: 30000)
--txpool.accountqueue value Maximum number of non-executable transaction slots permitted per account (default: 64)
--txpool.globalqueue value Maximum number of non-executable transaction slots for all accounts (default: 30000)
--txpool.lifetime value Maximum amount of time non-executable transaction are queued (default: 3h0m0s)
--txpool.trace.senders value Comma separared list of addresses, whoes transactions will traced in transaction pool with debug printing
--prune value Choose which ancient data delete from DB:
h - prune history (ChangeSets, HistoryIndices - used by historical state access, like eth_getStorageAt, eth_getBalanceAt, debug_traceTransaction, trace_block, trace_transaction, etc.)
r - prune receipts (Receipts, Logs, LogTopicIndex, LogAddressIndex - used by eth_getLogs and similar RPC methods)
t - prune transaction by it's hash index
c - prune call traces (used by trace_filter method)
Does delete data older than 90K blocks, --prune=h is shortcut for: --prune.h.older=90000.
Similarly, --prune=t is shortcut for: --prune.t.older=90000 and --prune=c is shortcut for: --prune.c.older=90000.
However, --prune=r means to prune receipts before the Beacon Chain genesis (Consensus Layer might need receipts after that).
If an item is NOT on the list - means NO pruning for this data.
Example: --prune=htc (default: "disabled")
--prune.h.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'h', then default is 90K) (default: 0)
--prune.r.older value Prune data older than this number of blocks from the tip of the chain (default: 0)
--prune.t.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 't', then default is 90K) (default: 0)
--prune.c.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'c', then default is 90K) (default: 0)
--prune.h.before value Prune data before this block (default: 0)
--prune.r.before value Prune data before this block (default: 0)
--prune.t.before value Prune data before this block (default: 0)
--prune.c.before value Prune data before this block (default: 0)
--batchSize value Batch size for the execution stage (default: "256M")
--blockDownloaderWindow value Outstanding limit of block bodies being downloaded (default: 32768)
--database.verbosity value Enabling internal db logs. Very high verbosity levels may require recompile db. Default: 2, means warning. (default: 2)
--private.api.addr value private api network address, for example: 127.0.0.1:9090, empty string means not to start the listener. do not expose to public network. serves remote database interface (default: "127.0.0.1:9090")
--private.api.ratelimit value Amount of requests server handle simultaneously - requests over this limit will wait. Increase it - if clients see 'request timeout' while server load is low - it means your 'hot data' is small or have much RAM. (default: 31872)
--etl.bufferSize value Buffer size for ETL operations. (default: "256MB")
--tls Enable TLS handshake
--tls.cert value Specify certificate
--tls.key value Specify key file
--tls.cacert value Specify certificate authority
--state.stream.disable Disable streaming of state changes from core to RPC daemon
--sync.loop.throttle value Sets the minimum time between sync loop starts (e.g. 1h30m, default is none)
--bad.block value Marks block with given hex string as bad and forces initial reorg before normal staged sync
--http HTTP-RPC server (enabled by default). Use --http=false to disable it
--http.addr value HTTP-RPC server listening interface (default: "localhost")
--http.port value HTTP-RPC server listening port (default: 8545)
--authrpc.addr value HTTP-RPC server listening interface for the Engine API (default: "localhost")
--authrpc.port value HTTP-RPC server listening port for the Engine API (default: 8551)
--authrpc.jwtsecret value Path to the token that ensures safe connection between CL and EL
--http.compression Enable compression over HTTP-RPC
--http.corsdomain value Comma separated list of domains from which to accept cross origin requests (browser enforced)
--http.vhosts value Comma separated list of virtual hostnames from which to accept requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--authrpc.vhosts value Comma separated list of virtual hostnames from which to accept Engine API requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--http.api value API's offered over the HTTP-RPC interface (default: "eth,erigon,engine")
--ws Enable the WS-RPC server
--ws.compression Enable compression over WebSocket
--http.trace Trace HTTP requests with INFO level
--state.cache value Amount of keys to store in StateCache (enabled if no --datadir set). Set 0 to disable StateCache. 1_000_000 keys ~ equal to 2Gb RAM (maybe we will add RAM accounting in future versions). (default: 1000000)
--rpc.batch.concurrency value Does limit amount of goroutines to process 1 batch request. Means 1 bach request can't overload server. 1 batch still can have unlimited amount of request (default: 2)
--rpc.streaming.disable Erigon has enalbed json streaming for some heavy endpoints (like trace_*). It's treadoff: greatly reduce amount of RAM (in some cases from 30GB to 30mb), but it produce invalid json format if error happened in the middle of streaming (because json is not streaming-friendly format)
--db.read.concurrency value Does limit amount of parallel db reads. Default: equal to GOMAXPROCS (or number of CPU) (default: 128)
--rpc.accessList value Specify granular (method-by-method) API allowlist
--trace.compat Bug for bug compatibility with OE for trace_ routines
--rpc.gascap value Sets a cap on gas that can be used in eth_call/estimateGas (default: 50000000)
--txpool.api.addr value txpool api network address, for example: 127.0.0.1:9090 (default: use value of --private.api.addr)
--trace.maxtraces value Sets a limit on traces that can be returned in trace_filter (default: 200)
--http.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--http.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--http.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If http.timeouts.idle is zero, the value of http.timeouts.read is used. (default: 2m0s)
--authrpc.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--authrpc.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--authrpc.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If authrpc.timeouts.idle is zero, the value of authrpc.timeouts.read is used. (default: 2m0s)
--rpc.evmtimeout value Maximum amount of time to wait for the answer from EVM call. (default: 5m0s)
--snap.keepblocks Keep ancient blocks in db (useful for debug)
--snap.stop Workaround to stop producing new snapshots, if you meet some snapshots-related critical bug
--db.pagesize value set mdbx pagesize on db creation: must be power of 2 and '256b <= pagesize <= 64kb'. default: equal to OperationSystem's pageSize (default: "4KB")
--torrent.port value port to listen and serve BitTorrent protocol (default: 42069)
--torrent.maxpeers value unused parameter (reserved for future use) (default: 100)
--torrent.conns.perfile value connections per file (default: 10)
--torrent.download.slots value amount of files to download in parallel. If network has enough seeders 1-3 slot enough, if network has lack of seeders increase to 5-7 (too big value will slow down everything). (default: 3)
--torrent.upload.rate value bytes per second, example: 32mb (default: "4mb")
--torrent.download.rate value bytes per second, example: 32mb (default: "16mb")
--torrent.verbosity value 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (must set --verbosity to equal or higher level and has defeault: 3) (default: 2)
--port value Network listening port (default: 30303)
--p2p.protocol value Version of eth p2p protocol (default: 66)
--nat value NAT port mapping mechanism (any|none|upnp|pmp|stun|extip:<IP>)
"" or "none" default - do not nat
"extip:77.12.33.4" will assume the local machine is reachable on the given IP
"any" uses the first auto-detected mechanism
"upnp" uses the Universal Plug and Play protocol
"pmp" uses NAT-PMP with an auto-detected gateway address
"pmp:192.168.0.1" uses NAT-PMP with the given gateway address
"stun" uses STUN to detect an external IP using a default server
"stun:<server>" uses STUN to detect an external IP using the given server (host:port)
--nodiscover Disables the peer discovery mechanism (manual peer addition)
--v5disc Enables the experimental RLPx V5 (Topic Discovery) mechanism
--netrestrict value Restricts network communication to the given IP networks (CIDR masks)
--nodekey value P2P node key file
--nodekeyhex value P2P node key as hex (for testing)
--discovery.dns value Sets DNS discovery entry points (use "" to disable DNS)
--bootnodes value Comma separated enode URLs for P2P discovery bootstrap
--staticpeers value Comma separated enode URLs to connect to
--trustedpeers value Comma separated enode URLs which are always allowed to connect, even above the peer limit
--maxpeers value Maximum number of network peers (network disabled if set to 0) (default: 100)
--chain value Name of the testnet to join (default: "mainnet")
--dev.period value Block period to use in developer mode (0 = mine only if transaction pending) (default: 0)
--vmdebug Record information useful for VM and contract debugging
--networkid value Explicitly set network id (integer)(For testnets: use --chain <testnet_name> instead) (default: 1)
--fakepow Disables proof-of-work verification
--gpo.blocks value Number of recent blocks to check for gas prices (default: 20)
--gpo.percentile value Suggested gas price is the given percentile of a set of recent transaction gas prices (default: 60)
--allow-insecure-unlock Allow insecure account unlocking when account-related RPCs are exposed by http
--metrics Enable metrics collection and reporting
--metrics.expensive Enable expensive metrics collection and reporting
--metrics.addr value Enable stand-alone metrics HTTP server listening interface (default: "127.0.0.1")
--metrics.port value Metrics HTTP server listening port (default: 6060)
--experimental.history.v3 (also known as Erigon3) Not recommended yet: Can't change this flag after node creation. New DB and Snapshots format of history allows: parallel blocks execution, get state as of given transaction without executing whole block.
--identity value Custom node name
--clique.checkpoint value number of blocks after which to save the vote snapshot to the database (default: 10)
--clique.snapshots value number of recent vote snapshots to keep in memory (default: 1024)
--clique.signatures value number of recent block signatures to keep in memory (default: 16384)
--clique.datadir value a path to clique db folder [<invalid Value>]
--watch-the-burn Enable WatchTheBurn stage to keep track of ETH issuance
--mine Enable mining
--proposer.disable Disables PoS proposer
--miner.notify value Comma separated HTTP URL list to notify of new work packages
--miner.gaslimit value Target gas limit for mined blocks (default: 30000000)
--miner.etherbase value Public address for block mining rewards (default: "0")
--miner.extradata value Block extra data set by the miner (default = client version)
--miner.noverify Disable remote sealing verification
--miner.sigfile value Private key to sign blocks with
--sentry.api.addr value comma separated sentry addresses '<host>:<port>,<host>:<port>'
--sentry.log-peer-info Log detailed peer info when a peer connects or disconnects. Enable to integrate with observer.
--downloader.api.addr value downloader address '<host>:<port>'
--no-downloader to disable downloader component
--downloader.verify verify snapshots on startup. it will not report founded problems but just re-download broken pieces
--healthcheck Enable grpc health check
--bor.heimdall value URL of Heimdall service (default: "http://localhost:1317")
--bor.withoutheimdall Run without Heimdall service (for testing purpose)
--ethstats value Reporting URL of a ethstats service (nodename:secret@host:port)
--override.terminaltotaldifficulty value Manually specify TerminalTotalDifficulty, overriding the bundled setting (default: <nil>) [<invalid Value>]
--override.mergeNetsplitBlock value Manually specify FORK_NEXT_VALUE (see EIP-3675), overriding the bundled setting (default: <nil>) [<invalid Value>]
--config value Sets erigon flags from YAML/TOML file
--verbosity value Logging verbosity: 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (default: 3)
--log.json Format logs with JSON
--pprof Enable the pprof HTTP server
--pprof.addr value pprof HTTP server listening interface (default: "127.0.0.1")
--pprof.port value pprof HTTP server listening port (default: 6060)
--pprof.cpuprofile value Write CPU profile to the given file
--trace value Write execution trace to the given file
--help, -h show help
--version, -v print the version
aws-cli/2.8.9 Python/3.9.11 Linux/5.15.0-1019-aws exe/x86_64.ubuntu.20 prompt/off
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/snapshots 599G 2.63T 599G /erigon/data/bsc/snapshots
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/parlia 116K 2.63T 116K /erigon/data/bsc/parlia
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/nodes 5.18M 2.63T 5.18M /erigon/data/bsc/nodes
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/chaindata 92.5G 2.63T 92.5G /erigon/data/bsc/chaindata
download: s3://public-blockchain-snapshots/bsc/erigon-nodes-folder-latest/eth66/mdbx.lck to ../erigon/data/bsc/nodes/eth66/mdbx.lck
download: s3://public-blockchain-snapshots/bsc/erigon-nodes-folder-latest/eth66/mdbx.dat to ../erigon/data/bsc/nodes/eth66/mdbx.dat
download: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/db/mdbx.lck to ../erigon/data/bsc/snapshots/db/mdbx.lck
download: s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/db/mdbx.dat to ../erigon/data/bsc/snapshots/db/mdbx.dat
root@ip-172-31-5-127:/bsc-archive-snapshot#
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
I manually run this command, it is downloading now, previously the data only has 4G
root@ip-172-31-5-127:/erigon/data/bsc/chaindata# ls -lh
total 2.2G
-rw-r--r-- 1 erigon erigon 4.0G Nov 7 02:36 mdbx.dat
-rw-r--r-- 1 erigon erigon 1004K Nov 7 05:04 mdbx.lck
root@ip-172-31-5-127:/erigon/data/bsc/chaindata# cd /
root@ip-172-31-5-127:/# s3pcp --requester-pays s3://public-blockchain-snapshots/bsc/erigon-16k-db-latest.mdbx.zstd \
| pv \ | pzstd -p $(nproc) -q -d -f -o /erigon/data/bsc/chaindata/mdbx.dat
153GiB 0:05:10 [ 517MiB/s] [ <=> ]
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Aaah, actually I needed stderr and stdout... But yeah that command might do what you need.
you can capture both stderr and stdout by using:
./build_archive_node.sh 2>&1 | tee /tmp/output.txt
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
root@ip-172-31-5-127:/bsc-archive-snapshot# ./build_archive_node.sh 2>&1 | tee /tmp/output.txt
+ [[ 0 -ne 0 ]]
+ install_prereq
+ set -euxo pipefail
+ apt update
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Fetched 114 kB in 1s (134 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
60 packages can be upgraded. Run 'apt list --upgradable' to see them.
+ DEBIAN_FRONTEND=noninteractive
+ apt install -y zfsutils-linux unzip pv clang-12 make jq python3-boto3 super
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
make is already the newest version (4.2.1-1.2).
pv is already the newest version (1.6.6-1).
python3-boto3 is already the newest version (1.9.253-1).
super is already the newest version (3.30.1-1).
unzip is already the newest version (6.0-25ubuntu1.1).
zfsutils-linux is already the newest version (0.8.3-1ubuntu12.14).
clang-12 is already the newest version (1:12.0.0-3ubuntu1~20.04.5).
jq is already the newest version (1.6-1ubuntu0.20.04.1).
0 upgraded, 0 newly installed, 0 to remove and 60 not upgraded.
++ which clang-12
+ ln -s /usr/bin/clang-12 /usr/bin/cc
ln: failed to create symbolic link '/usr/bin/cc': File exists
+ true
+ snap install --classic go
snap "go" is already installed, see 'snap help refresh'
+ cargo --version
+ setup_drives
+ set -euxo pipefail
+ zfs list tank
NAME USED AVAIL REFER MOUNTPOINT
tank 1.13T 2.17T 96K none
+ return
+ install_zstd
+ set -euxo pipefail
+ pzstd --help
+ install_aws_cli
+ install_s3pcp
+ set -euxo pipefail
+ aws --version
+ set -euxo pipefail
+ s3pcp --help
+ install_putils
+ safe_wait
+ BACKGROUND_PIDS=($(jobs -p))
+ install_erigon
+ set -euxo pipefail
+ psplit --help
+ set -euxo pipefail
+ erigon --help
++ jobs -p
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387628
psplit 0.1.0
Takes a file or stdin and splits it by -b bytes and sends them to the
OUTPUT_COMMAND in parallel limited by -p concurrent jobs.
Example:
$ printf '012345678901234567890123' | psplit -b 10 sh -c 'cat > /tmp/psplit_$SEQ; echo >>
/tmp/psplit_$SEQ'
Will result in 3 files:
/tmp/psplit_0 = "0123456789
"
/tmp/psplit_1 = "0123456789
"
/tmp/psplit_2 = "0123
"
USAGE:
psplit [OPTIONS] <OUTPUT_COMMAND>
ARGS:
<OUTPUT_COMMAND> Command to run on each output. An environmental variable "SEQ" will be
set containing the sequence number of the slice. Data will be sent to
stdin of command
OPTIONS:
-b, --bytes <BYTES>
Number of bytes per output file [default: 1073741824]
-h, --help
Print help information
-i, --input-file <INPUT_FILE>
Input file to split. If not set, uses stdin
-p, --parallel-count <PARALLEL_COUNT>
Number of commands allowed to run in parallel [default: 16]
-V, --version
Print version information
Usage:
pzstd [args] [FILE(s)]
Parallel ZSTD options:
-p, --processes # : number of threads to use for (de)compression (default:<numcpus>)
ZSTD options:
+ pjoin --help
-# : # compression level (1-19, default:3)
-d, --decompress : decompression
-o file : result stored into `file` (only if 1 input file)
-f, --force : overwrite output without prompting, (de)compress links
--rm : remove source file(s) after successful (de)compression
-k, --keep : preserve source file(s) (default)
-h, --help : display help and exit
-V, --version : display version number and exit
-v, --verbose : verbose mode; specify multiple times to increase log level (default:2)
-q, --quiet : suppress warnings; specify twice to suppress errors too
-c, --stdout : write to standard output (even if it is the console)
-r : operate recursively on directories
--ultra : enable levels beyond 19, up to 22 (requires more memory)
-C, --check : integrity check (default)
--no-check : no integrity check
-t, --test : test compressed file integrity
-- : all arguments after "--" are treated as files
+ return
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387629
pjoin 0.1.0
Takes stdin commands split by new lines and executes them in parallel and
prints stdout in order based on the order they are in stdin.
Example:
$ pjoin <<'EOT'
sh -c 'printf foo; sleep 1'
printf bar
EOT
Will result in:
'foobar' (without quotes)
USAGE:
pjoin [OPTIONS] [OUTPUT_FILE]
ARGS:
<OUTPUT_FILE> Path to write file. Prints to stdout if not set. Using a file can be faster
than stdout
OPTIONS:
-b, --buffer-size <BUFFER_SIZE>
Size in bytes of the stdout buffer for reach command [default: 1073741824]
-h, --help
Print help information
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: 16]
-V, --version
Print version information
+ return
Downloads data from s3 and sends it to stdout very fast and on a pinned version.
Many concurrent connections to s3 are opened and different chunks of the file
are downloaded in parallel and stiched together using the `pjoin` utility.
USAGE:
s3pcp [OPTIONS] [S3_PATH]
ARGS:
S3_PATH A path to an s3 object. Format: 's3://{bucket}/{key}'
OPTIONS:
--requester-pays
If the account downloading is requesting to be the payer for
the request.
--region <REGION>
The region the request should be sent to.
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: based on computer resources]
-h, --help
Print help information
+ return
erigon [global options] command [command options] [arguments...]
VERSION:
2.28.1-stable-9711af6d
COMMANDS:
init Bootstrap and initialize a new genesis block
import Import a blockchain file
snapshots
help Shows a list of commands or help for one command
GLOBAL OPTIONS:
--datadir value Data directory for the databases (default: "/root/.local/share/erigon") [<invalid Value>]
--ethash.dagdir value Directory to store the ethash mining DAGs (default: "/root/.local/share/erigon-ethash") [<invalid Value>]
--snapshots Default: use snapshots "true" for BSC, Mainnet and Goerli. use snapshots "false" in all other cases
--experimental.lightclient enables experimental CL lightclient.
--txpool.disable experimental external pool and block producer, see ./cmd/txpool/readme.md for more info. Disabling internal txpool and block producer.
--txpool.locals value Comma separated accounts to treat as locals (no flush, priority inclusion)
--txpool.nolocals Disables price exemptions for locally submitted transactions
--txpool.pricelimit value Minimum gas price (fee cap) limit to enforce for acceptance into the pool (default: 1)
--txpool.pricebump value Price bump percentage to replace an already existing transaction (default: 10)
--txpool.accountslots value Minimum number of executable transaction slots guaranteed per account (default: 16)
--txpool.globalslots value Maximum number of executable transaction slots for all accounts (default: 10000)
--txpool.globalbasefeeslots value Maximum number of non-executable transactions where only not enough baseFee (default: 30000)
--txpool.accountqueue value Maximum number of non-executable transaction slots permitted per account (default: 64)
--txpool.globalqueue value Maximum number of non-executable transaction slots for all accounts (default: 30000)
--txpool.lifetime value Maximum amount of time non-executable transaction are queued (default: 3h0m0s)
--txpool.trace.senders value Comma separared list of addresses, whoes transactions will traced in transaction pool with debug printing
--prune value Choose which ancient data delete from DB:
h - prune history (ChangeSets, HistoryIndices - used by historical state access, like eth_getStorageAt, eth_getBalanceAt, debug_traceTransaction, trace_block, trace_transaction, etc.)
r - prune receipts (Receipts, Logs, LogTopicIndex, LogAddressIndex - used by eth_getLogs and similar RPC methods)
t - prune transaction by it's hash index
c - prune call traces (used by trace_filter method)
Does delete data older than 90K blocks, --prune=h is shortcut for: --prune.h.older=90000.
Similarly, --prune=t is shortcut for: --prune.t.older=90000 and --prune=c is shortcut for: --prune.c.older=90000.
However, --prune=r means to prune receipts before the Beacon Chain genesis (Consensus Layer might need receipts after that).
If an item is NOT on the list - means NO pruning for this data.
Example: --prune=htc (default: "disabled")
--prune.h.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'h', then default is 90K) (default: 0)
--prune.r.older value Prune data older than this number of blocks from the tip of the chain (default: 0)
--prune.t.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 't', then default is 90K) (default: 0)
--prune.c.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'c', then default is 90K) (default: 0)
--prune.h.before value Prune data before this block (default: 0)
--prune.r.before value Prune data before this block (default: 0)
--prune.t.before value Prune data before this block (default: 0)
--prune.c.before value Prune data before this block (default: 0)
--batchSize value Batch size for the execution stage (default: "256M")
--blockDownloaderWindow value Outstanding limit of block bodies being downloaded (default: 32768)
--database.verbosity value Enabling internal db logs. Very high verbosity levels may require recompile db. Default: 2, means warning. (default: 2)
--private.api.addr value private api network address, for example: 127.0.0.1:9090, empty string means not to start the listener. do not expose to public network. serves remote database interface (default: "127.0.0.1:9090")
--private.api.ratelimit value Amount of requests server handle simultaneously - requests over this limit will wait. Increase it - if clients see 'request timeout' while server load is low - it means your 'hot data' is small or have much RAM. (default: 31872)
--etl.bufferSize value Buffer size for ETL operations. (default: "256MB")
--tls Enable TLS handshake
--tls.cert value Specify certificate
--tls.key value Specify key file
--tls.cacert value Specify certificate authority
--state.stream.disable Disable streaming of state changes from core to RPC daemon
--sync.loop.throttle value Sets the minimum time between sync loop starts (e.g. 1h30m, default is none)
--bad.block value Marks block with given hex string as bad and forces initial reorg before normal staged sync
--http HTTP-RPC server (enabled by default). Use --http=false to disable it
--http.addr value HTTP-RPC server listening interface (default: "localhost")
--http.port value HTTP-RPC server listening port (default: 8545)
--authrpc.addr value HTTP-RPC server listening interface for the Engine API (default: "localhost")
--authrpc.port value HTTP-RPC server listening port for the Engine API (default: 8551)
--authrpc.jwtsecret value Path to the token that ensures safe connection between CL and EL
--http.compression Enable compression over HTTP-RPC
--http.corsdomain value Comma separated list of domains from which to accept cross origin requests (browser enforced)
--http.vhosts value Comma separated list of virtual hostnames from which to accept requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--authrpc.vhosts value Comma separated list of virtual hostnames from which to accept Engine API requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--http.api value API's offered over the HTTP-RPC interface (default: "eth,erigon,engine")
--ws Enable the WS-RPC server
--ws.compression Enable compression over WebSocket
--http.trace Trace HTTP requests with INFO level
--state.cache value Amount of keys to store in StateCache (enabled if no --datadir set). Set 0 to disable StateCache. 1_000_000 keys ~ equal to 2Gb RAM (maybe we will add RAM accounting in future versions). (default: 1000000)
--rpc.batch.concurrency value Does limit amount of goroutines to process 1 batch request. Means 1 bach request can't overload server. 1 batch still can have unlimited amount of request (default: 2)
--rpc.streaming.disable Erigon has enalbed json streaming for some heavy endpoints (like trace_*). It's treadoff: greatly reduce amount of RAM (in some cases from 30GB to 30mb), but it produce invalid json format if error happened in the middle of streaming (because json is not streaming-friendly format)
--db.read.concurrency value Does limit amount of parallel db reads. Default: equal to GOMAXPROCS (or number of CPU) (default: 128)
--rpc.accessList value Specify granular (method-by-method) API allowlist
--trace.compat Bug for bug compatibility with OE for trace_ routines
--rpc.gascap value Sets a cap on gas that can be used in eth_call/estimateGas (default: 50000000)
--txpool.api.addr value txpool api network address, for example: 127.0.0.1:9090 (default: use value of --private.api.addr)
--trace.maxtraces value Sets a limit on traces that can be returned in trace_filter (default: 200)
--http.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--http.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--http.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If http.timeouts.idle is zero, the value of http.timeouts.read is used. (default: 2m0s)
--authrpc.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--authrpc.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--authrpc.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If authrpc.timeouts.idle is zero, the value of authrpc.timeouts.read is used. (default: 2m0s)
--rpc.evmtimeout value Maximum amount of time to wait for the answer from EVM call. (default: 5m0s)
--snap.keepblocks Keep ancient blocks in db (useful for debug)
--snap.stop Workaround to stop producing new snapshots, if you meet some snapshots-related critical bug
--db.pagesize value set mdbx pagesize on db creation: must be power of 2 and '256b <= pagesize <= 64kb'. default: equal to OperationSystem's pageSize (default: "4KB")
--torrent.port value port to listen and serve BitTorrent protocol (default: 42069)
--torrent.maxpeers value unused parameter (reserved for future use) (default: 100)
--torrent.conns.perfile value connections per file (default: 10)
--torrent.download.slots value amount of files to download in parallel. If network has enough seeders 1-3 slot enough, if network has lack of seeders increase to 5-7 (too big value will slow down everything). (default: 3)
--torrent.upload.rate value bytes per second, example: 32mb (default: "4mb")
--torrent.download.rate value bytes per second, example: 32mb (default: "16mb")
--torrent.verbosity value 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (must set --verbosity to equal or higher level and has defeault: 3) (default: 2)
--port value Network listening port (default: 30303)
--p2p.protocol value Version of eth p2p protocol (default: 66)
--nat value NAT port mapping mechanism (any|none|upnp|pmp|stun|extip:<IP>)
"" or "none" default - do not nat
"extip:77.12.33.4" will assume the local machine is reachable on the given IP
"any" uses the first auto-detected mechanism
"upnp" uses the Universal Plug and Play protocol
"pmp" uses NAT-PMP with an auto-detected gateway address
"pmp:192.168.0.1" uses NAT-PMP with the given gateway address
"stun" uses STUN to detect an external IP using a default server
"stun:<server>" uses STUN to detect an external IP using the given server (host:port)
--nodiscover Disables the peer discovery mechanism (manual peer addition)
--v5disc Enables the experimental RLPx V5 (Topic Discovery) mechanism
--netrestrict value Restricts network communication to the given IP networks (CIDR masks)
--nodekey value P2P node key file
--nodekeyhex value P2P node key as hex (for testing)
--discovery.dns value Sets DNS discovery entry points (use "" to disable DNS)
--bootnodes value Comma separated enode URLs for P2P discovery bootstrap
--staticpeers value Comma separated enode URLs to connect to
--trustedpeers value Comma separated enode URLs which are always allowed to connect, even above the peer limit
--maxpeers value Maximum number of network peers (network disabled if set to 0) (default: 100)
--chain value Name of the testnet to join (default: "mainnet")
--dev.period value Block period to use in developer mode (0 = mine only if transaction pending) (default: 0)
--vmdebug Record information useful for VM and contract debugging
--networkid value Explicitly set network id (integer)(For testnets: use --chain <testnet_name> instead) (default: 1)
--fakepow Disables proof-of-work verification
--gpo.blocks value Number of recent blocks to check for gas prices (default: 20)
--gpo.percentile value Suggested gas price is the given percentile of a set of recent transaction gas prices (default: 60)
--allow-insecure-unlock Allow insecure account unlocking when account-related RPCs are exposed by http
--metrics Enable metrics collection and reporting
--metrics.expensive Enable expensive metrics collection and reporting
--metrics.addr value Enable stand-alone metrics HTTP server listening interface (default: "127.0.0.1")
--metrics.port value Metrics HTTP server listening port (default: 6060)
--experimental.history.v3 (also known as Erigon3) Not recommended yet: Can't change this flag after node creation. New DB and Snapshots format of history allows: parallel blocks execution, get state as of given transaction without executing whole block.
--identity value Custom node name
--clique.checkpoint value number of blocks after which to save the vote snapshot to the database (default: 10)
--clique.snapshots value number of recent vote snapshots to keep in memory (default: 1024)
--clique.signatures value number of recent block signatures to keep in memory (default: 16384)
--clique.datadir value a path to clique db folder [<invalid Value>]
--watch-the-burn Enable WatchTheBurn stage to keep track of ETH issuance
--mine Enable mining
--proposer.disable Disables PoS proposer
--miner.notify value Comma separated HTTP URL list to notify of new work packages
--miner.gaslimit value Target gas limit for mined blocks (default: 30000000)
--miner.etherbase value Public address for block mining rewards (default: "0")
--miner.extradata value Block extra data set by the miner (default = client version)
--miner.noverify Disable remote sealing verification
--miner.sigfile value Private key to sign blocks with
--sentry.api.addr value comma separated sentry addresses '<host>:<port>,<host>:<port>'
--sentry.log-peer-info Log detailed peer info when a peer connects or disconnects. Enable to integrate with observer.
--downloader.api.addr value downloader address '<host>:<port>'
--no-downloader to disable downloader component
--downloader.verify verify snapshots on startup. it will not report founded problems but just re-download broken pieces
--healthcheck Enable grpc health check
--bor.heimdall value URL of Heimdall service (default: "http://localhost:1317")
--bor.withoutheimdall Run without Heimdall service (for testing purpose)
--ethstats value Reporting URL of a ethstats service (nodename:secret@host:port)
--override.terminaltotaldifficulty value Manually specify TerminalTotalDifficulty, overriding the bundled setting (default: <nil>) [<invalid Value>]
--override.mergeNetsplitBlock value Manually specify FORK_NEXT_VALUE (see EIP-3675), overriding the bundled setting (default: <nil>) [<invalid Value>]
--config value Sets erigon flags from YAML/TOML file
--verbosity value Logging verbosity: 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (default: 3)
--log.json Format logs with JSON
--pprof Enable the pprof HTTP server
--pprof.addr value pprof HTTP server listening interface (default: "127.0.0.1")
--pprof.port value pprof HTTP server listening port (default: 6060)
--pprof.cpuprofile value Write CPU profile to the given file
--trace value Write execution trace to the given file
--help, -h show help
--version, -v print the version
+ return
aws-cli/2.8.9 Python/3.9.11 Linux/5.15.0-1019-aws exe/x86_64.ubuntu.20 prompt/off
+ return
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387630
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387632
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387633
+ prepare_zfs_datasets
+ set -euxo pipefail
+ zfs create -o mountpoint=/erigon/data tank/erigon_data
cannot create 'tank/erigon_data': dataset already exists
+ true
+ zfs create -o mountpoint=/erigon/data/bsc tank/erigon_data/bsc
cannot create 'tank/erigon_data/bsc': dataset already exists
+ true
+ download_snapshots
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/snapshots
+ download_nodes
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/nodes
+ download_parlia
+ set -euxo pipefail
+ safe_wait
+ zfs list tank/erigon_data/bsc/parlia
+ BACKGROUND_PIDS=($(jobs -p))
+ download_database_file
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/chaindata
++ jobs -p
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387943
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/nodes 5.57M 2.17T 5.57M /erigon/data/bsc/nodes
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/parlia 116K 2.17T 116K /erigon/data/bsc/parlia
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/snapshots 599G 2.17T 599G /erigon/data/bsc/snapshots
+ aws s3 sync --request-payer requester s3://public-blockchain-snapshots/bsc/erigon-nodes-folder-latest/ /erigon/data/bsc/nodes/
+ mkdir -p /erigon/data/bsc/snapshots/
+ aws s3 cp --request-payer requester s3://public-blockchain-snapshots/bsc/parlia-db-latest.mdbx.zstd -
+ pzstd -d -o /erigon/data/bsc/parlia/mdbx.dat
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/chaindata 560G 2.17T 560G /erigon/data/bsc/chaindata
+ return
+ aws s3 sync --request-payer requester s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/ /erigon/data/bsc/snapshots/
pzstd: /erigon/data/bsc/parlia/mdbx.dat already exists; do you wish to overwrite (y/n) ? pzstd: -: Not overwritten.
download failed: s3://public-blockchain-snapshots/bsc/parlia-db-latest.mdbx.zstd to - [Errno 32] Broken pipe
+ true
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387944
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387945
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387947
+ prepare_erigon
+ set -euxo pipefail
+ useradd erigon
useradd: user 'erigon' already exists
+ true
+ chown -R erigon:erigon /erigon/data/
+ systemctl stop erigon-bsc
+ echo '[Unit]
Description=Erigon BSC daemon
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=1
User=erigon
ExecStart=/erigon/start_erigon_service.sh
[Install]
WantedBy=multi-user.target
'
+ echo '#!/bin/bash'
+ [[ '' == \1 ]]
+ echo -n 'exec erigon --chain bsc --snapshots=true --db.pagesize=16k --datadir=/erigon/data/bsc --txpool.disable'
+ [[ '' == \1 ]]
+ chmod +x /erigon/start_erigon_service.sh
+ systemctl daemon-reload
+ systemctl enable erigon-bsc
+ run_erigon
+ set -euxo pipefail
+ systemctl start erigon-bsc
+ add_create_snapshot_script
+ set -euxo pipefail
+ cat
+ chmod 0744 /erigon/create-bsc-snapshot-and-shutdown.sh
+ chown root:root /erigon/create-bsc-snapshot-and-shutdown.sh
+ echo 'create-bsc-snapshot-and-shutdown /erigon/create-bsc-snapshot-and-shutdown.sh uid=root erigon'
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot#
root@ip-172-31-5-127:/bsc-archive-snapshot# cat /tmp/output.txt
+ [[ 0 -ne 0 ]]
+ install_prereq
+ set -euxo pipefail
+ apt update
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu focal-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Fetched 114 kB in 1s (134 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
60 packages can be upgraded. Run 'apt list --upgradable' to see them.
+ DEBIAN_FRONTEND=noninteractive
+ apt install -y zfsutils-linux unzip pv clang-12 make jq python3-boto3 super
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
make is already the newest version (4.2.1-1.2).
pv is already the newest version (1.6.6-1).
python3-boto3 is already the newest version (1.9.253-1).
super is already the newest version (3.30.1-1).
unzip is already the newest version (6.0-25ubuntu1.1).
zfsutils-linux is already the newest version (0.8.3-1ubuntu12.14).
clang-12 is already the newest version (1:12.0.0-3ubuntu1~20.04.5).
jq is already the newest version (1.6-1ubuntu0.20.04.1).
0 upgraded, 0 newly installed, 0 to remove and 60 not upgraded.
++ which clang-12
+ ln -s /usr/bin/clang-12 /usr/bin/cc
ln: failed to create symbolic link '/usr/bin/cc': File exists
+ true
+ snap install --classic go
snap "go" is already installed, see 'snap help refresh'
+ cargo --version
+ setup_drives
+ set -euxo pipefail
+ zfs list tank
NAME USED AVAIL REFER MOUNTPOINT
tank 1.13T 2.17T 96K none
+ return
+ install_zstd
+ set -euxo pipefail
+ pzstd --help
+ install_aws_cli
+ install_s3pcp
+ set -euxo pipefail
+ aws --version
+ set -euxo pipefail
+ s3pcp --help
+ install_putils
+ safe_wait
+ BACKGROUND_PIDS=($(jobs -p))
+ install_erigon
+ set -euxo pipefail
+ psplit --help
+ set -euxo pipefail
+ erigon --help
++ jobs -p
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387628
psplit 0.1.0
Takes a file or stdin and splits it by -b bytes and sends them to the
OUTPUT_COMMAND in parallel limited by -p concurrent jobs.
Example:
$ printf '012345678901234567890123' | psplit -b 10 sh -c 'cat > /tmp/psplit_$SEQ; echo >>
/tmp/psplit_$SEQ'
Will result in 3 files:
/tmp/psplit_0 = "0123456789
"
/tmp/psplit_1 = "0123456789
"
/tmp/psplit_2 = "0123
"
USAGE:
psplit [OPTIONS] <OUTPUT_COMMAND>
ARGS:
<OUTPUT_COMMAND> Command to run on each output. An environmental variable "SEQ" will be
set containing the sequence number of the slice. Data will be sent to
stdin of command
OPTIONS:
-b, --bytes <BYTES>
Number of bytes per output file [default: 1073741824]
-h, --help
Print help information
-i, --input-file <INPUT_FILE>
Input file to split. If not set, uses stdin
-p, --parallel-count <PARALLEL_COUNT>
Number of commands allowed to run in parallel [default: 16]
-V, --version
Print version information
Usage:
pzstd [args] [FILE(s)]
Parallel ZSTD options:
-p, --processes # : number of threads to use for (de)compression (default:<numcpus>)
ZSTD options:
+ pjoin --help
-# : # compression level (1-19, default:3)
-d, --decompress : decompression
-o file : result stored into `file` (only if 1 input file)
-f, --force : overwrite output without prompting, (de)compress links
--rm : remove source file(s) after successful (de)compression
-k, --keep : preserve source file(s) (default)
-h, --help : display help and exit
-V, --version : display version number and exit
-v, --verbose : verbose mode; specify multiple times to increase log level (default:2)
-q, --quiet : suppress warnings; specify twice to suppress errors too
-c, --stdout : write to standard output (even if it is the console)
-r : operate recursively on directories
--ultra : enable levels beyond 19, up to 22 (requires more memory)
-C, --check : integrity check (default)
--no-check : no integrity check
-t, --test : test compressed file integrity
-- : all arguments after "--" are treated as files
+ return
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387629
pjoin 0.1.0
Takes stdin commands split by new lines and executes them in parallel and
prints stdout in order based on the order they are in stdin.
Example:
$ pjoin <<'EOT'
sh -c 'printf foo; sleep 1'
printf bar
EOT
Will result in:
'foobar' (without quotes)
USAGE:
pjoin [OPTIONS] [OUTPUT_FILE]
ARGS:
<OUTPUT_FILE> Path to write file. Prints to stdout if not set. Using a file can be faster
than stdout
OPTIONS:
-b, --buffer-size <BUFFER_SIZE>
Size in bytes of the stdout buffer for reach command [default: 1073741824]
-h, --help
Print help information
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: 16]
-V, --version
Print version information
+ return
Downloads data from s3 and sends it to stdout very fast and on a pinned version.
Many concurrent connections to s3 are opened and different chunks of the file
are downloaded in parallel and stiched together using the `pjoin` utility.
USAGE:
s3pcp [OPTIONS] [S3_PATH]
ARGS:
S3_PATH A path to an s3 object. Format: 's3://{bucket}/{key}'
OPTIONS:
--requester-pays
If the account downloading is requesting to be the payer for
the request.
--region <REGION>
The region the request should be sent to.
-p, --parallel-count <PARALLEL_COUNT>
Number of commands to run in parallel [default: based on computer resources]
-h, --help
Print help information
+ return
erigon [global options] command [command options] [arguments...]
VERSION:
2.28.1-stable-9711af6d
COMMANDS:
init Bootstrap and initialize a new genesis block
import Import a blockchain file
snapshots
help Shows a list of commands or help for one command
GLOBAL OPTIONS:
--datadir value Data directory for the databases (default: "/root/.local/share/erigon") [<invalid Value>]
--ethash.dagdir value Directory to store the ethash mining DAGs (default: "/root/.local/share/erigon-ethash") [<invalid Value>]
--snapshots Default: use snapshots "true" for BSC, Mainnet and Goerli. use snapshots "false" in all other cases
--experimental.lightclient enables experimental CL lightclient.
--txpool.disable experimental external pool and block producer, see ./cmd/txpool/readme.md for more info. Disabling internal txpool and block producer.
--txpool.locals value Comma separated accounts to treat as locals (no flush, priority inclusion)
--txpool.nolocals Disables price exemptions for locally submitted transactions
--txpool.pricelimit value Minimum gas price (fee cap) limit to enforce for acceptance into the pool (default: 1)
--txpool.pricebump value Price bump percentage to replace an already existing transaction (default: 10)
--txpool.accountslots value Minimum number of executable transaction slots guaranteed per account (default: 16)
--txpool.globalslots value Maximum number of executable transaction slots for all accounts (default: 10000)
--txpool.globalbasefeeslots value Maximum number of non-executable transactions where only not enough baseFee (default: 30000)
--txpool.accountqueue value Maximum number of non-executable transaction slots permitted per account (default: 64)
--txpool.globalqueue value Maximum number of non-executable transaction slots for all accounts (default: 30000)
--txpool.lifetime value Maximum amount of time non-executable transaction are queued (default: 3h0m0s)
--txpool.trace.senders value Comma separared list of addresses, whoes transactions will traced in transaction pool with debug printing
--prune value Choose which ancient data delete from DB:
h - prune history (ChangeSets, HistoryIndices - used by historical state access, like eth_getStorageAt, eth_getBalanceAt, debug_traceTransaction, trace_block, trace_transaction, etc.)
r - prune receipts (Receipts, Logs, LogTopicIndex, LogAddressIndex - used by eth_getLogs and similar RPC methods)
t - prune transaction by it's hash index
c - prune call traces (used by trace_filter method)
Does delete data older than 90K blocks, --prune=h is shortcut for: --prune.h.older=90000.
Similarly, --prune=t is shortcut for: --prune.t.older=90000 and --prune=c is shortcut for: --prune.c.older=90000.
However, --prune=r means to prune receipts before the Beacon Chain genesis (Consensus Layer might need receipts after that).
If an item is NOT on the list - means NO pruning for this data.
Example: --prune=htc (default: "disabled")
--prune.h.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'h', then default is 90K) (default: 0)
--prune.r.older value Prune data older than this number of blocks from the tip of the chain (default: 0)
--prune.t.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 't', then default is 90K) (default: 0)
--prune.c.older value Prune data older than this number of blocks from the tip of the chain (if --prune flag has 'c', then default is 90K) (default: 0)
--prune.h.before value Prune data before this block (default: 0)
--prune.r.before value Prune data before this block (default: 0)
--prune.t.before value Prune data before this block (default: 0)
--prune.c.before value Prune data before this block (default: 0)
--batchSize value Batch size for the execution stage (default: "256M")
--blockDownloaderWindow value Outstanding limit of block bodies being downloaded (default: 32768)
--database.verbosity value Enabling internal db logs. Very high verbosity levels may require recompile db. Default: 2, means warning. (default: 2)
--private.api.addr value private api network address, for example: 127.0.0.1:9090, empty string means not to start the listener. do not expose to public network. serves remote database interface (default: "127.0.0.1:9090")
--private.api.ratelimit value Amount of requests server handle simultaneously - requests over this limit will wait. Increase it - if clients see 'request timeout' while server load is low - it means your 'hot data' is small or have much RAM. (default: 31872)
--etl.bufferSize value Buffer size for ETL operations. (default: "256MB")
--tls Enable TLS handshake
--tls.cert value Specify certificate
--tls.key value Specify key file
--tls.cacert value Specify certificate authority
--state.stream.disable Disable streaming of state changes from core to RPC daemon
--sync.loop.throttle value Sets the minimum time between sync loop starts (e.g. 1h30m, default is none)
--bad.block value Marks block with given hex string as bad and forces initial reorg before normal staged sync
--http HTTP-RPC server (enabled by default). Use --http=false to disable it
--http.addr value HTTP-RPC server listening interface (default: "localhost")
--http.port value HTTP-RPC server listening port (default: 8545)
--authrpc.addr value HTTP-RPC server listening interface for the Engine API (default: "localhost")
--authrpc.port value HTTP-RPC server listening port for the Engine API (default: 8551)
--authrpc.jwtsecret value Path to the token that ensures safe connection between CL and EL
--http.compression Enable compression over HTTP-RPC
--http.corsdomain value Comma separated list of domains from which to accept cross origin requests (browser enforced)
--http.vhosts value Comma separated list of virtual hostnames from which to accept requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--authrpc.vhosts value Comma separated list of virtual hostnames from which to accept Engine API requests (server enforced). Accepts '*' wildcard. (default: "localhost")
--http.api value API's offered over the HTTP-RPC interface (default: "eth,erigon,engine")
--ws Enable the WS-RPC server
--ws.compression Enable compression over WebSocket
--http.trace Trace HTTP requests with INFO level
--state.cache value Amount of keys to store in StateCache (enabled if no --datadir set). Set 0 to disable StateCache. 1_000_000 keys ~ equal to 2Gb RAM (maybe we will add RAM accounting in future versions). (default: 1000000)
--rpc.batch.concurrency value Does limit amount of goroutines to process 1 batch request. Means 1 bach request can't overload server. 1 batch still can have unlimited amount of request (default: 2)
--rpc.streaming.disable Erigon has enalbed json streaming for some heavy endpoints (like trace_*). It's treadoff: greatly reduce amount of RAM (in some cases from 30GB to 30mb), but it produce invalid json format if error happened in the middle of streaming (because json is not streaming-friendly format)
--db.read.concurrency value Does limit amount of parallel db reads. Default: equal to GOMAXPROCS (or number of CPU) (default: 128)
--rpc.accessList value Specify granular (method-by-method) API allowlist
--trace.compat Bug for bug compatibility with OE for trace_ routines
--rpc.gascap value Sets a cap on gas that can be used in eth_call/estimateGas (default: 50000000)
--txpool.api.addr value txpool api network address, for example: 127.0.0.1:9090 (default: use value of --private.api.addr)
--trace.maxtraces value Sets a limit on traces that can be returned in trace_filter (default: 200)
--http.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--http.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--http.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If http.timeouts.idle is zero, the value of http.timeouts.read is used. (default: 2m0s)
--authrpc.timeouts.read value Maximum duration for reading the entire request, including the body. (default: 30s)
--authrpc.timeouts.write value Maximum duration before timing out writes of the response. It is reset whenever a new request's header is read. (default: 30m0s)
--authrpc.timeouts.idle value Maximum amount of time to wait for the next request when keep-alives are enabled. If authrpc.timeouts.idle is zero, the value of authrpc.timeouts.read is used. (default: 2m0s)
--rpc.evmtimeout value Maximum amount of time to wait for the answer from EVM call. (default: 5m0s)
--snap.keepblocks Keep ancient blocks in db (useful for debug)
--snap.stop Workaround to stop producing new snapshots, if you meet some snapshots-related critical bug
--db.pagesize value set mdbx pagesize on db creation: must be power of 2 and '256b <= pagesize <= 64kb'. default: equal to OperationSystem's pageSize (default: "4KB")
--torrent.port value port to listen and serve BitTorrent protocol (default: 42069)
--torrent.maxpeers value unused parameter (reserved for future use) (default: 100)
--torrent.conns.perfile value connections per file (default: 10)
--torrent.download.slots value amount of files to download in parallel. If network has enough seeders 1-3 slot enough, if network has lack of seeders increase to 5-7 (too big value will slow down everything). (default: 3)
--torrent.upload.rate value bytes per second, example: 32mb (default: "4mb")
--torrent.download.rate value bytes per second, example: 32mb (default: "16mb")
--torrent.verbosity value 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (must set --verbosity to equal or higher level and has defeault: 3) (default: 2)
--port value Network listening port (default: 30303)
--p2p.protocol value Version of eth p2p protocol (default: 66)
--nat value NAT port mapping mechanism (any|none|upnp|pmp|stun|extip:<IP>)
"" or "none" default - do not nat
"extip:77.12.33.4" will assume the local machine is reachable on the given IP
"any" uses the first auto-detected mechanism
"upnp" uses the Universal Plug and Play protocol
"pmp" uses NAT-PMP with an auto-detected gateway address
"pmp:192.168.0.1" uses NAT-PMP with the given gateway address
"stun" uses STUN to detect an external IP using a default server
"stun:<server>" uses STUN to detect an external IP using the given server (host:port)
--nodiscover Disables the peer discovery mechanism (manual peer addition)
--v5disc Enables the experimental RLPx V5 (Topic Discovery) mechanism
--netrestrict value Restricts network communication to the given IP networks (CIDR masks)
--nodekey value P2P node key file
--nodekeyhex value P2P node key as hex (for testing)
--discovery.dns value Sets DNS discovery entry points (use "" to disable DNS)
--bootnodes value Comma separated enode URLs for P2P discovery bootstrap
--staticpeers value Comma separated enode URLs to connect to
--trustedpeers value Comma separated enode URLs which are always allowed to connect, even above the peer limit
--maxpeers value Maximum number of network peers (network disabled if set to 0) (default: 100)
--chain value Name of the testnet to join (default: "mainnet")
--dev.period value Block period to use in developer mode (0 = mine only if transaction pending) (default: 0)
--vmdebug Record information useful for VM and contract debugging
--networkid value Explicitly set network id (integer)(For testnets: use --chain <testnet_name> instead) (default: 1)
--fakepow Disables proof-of-work verification
--gpo.blocks value Number of recent blocks to check for gas prices (default: 20)
--gpo.percentile value Suggested gas price is the given percentile of a set of recent transaction gas prices (default: 60)
--allow-insecure-unlock Allow insecure account unlocking when account-related RPCs are exposed by http
--metrics Enable metrics collection and reporting
--metrics.expensive Enable expensive metrics collection and reporting
--metrics.addr value Enable stand-alone metrics HTTP server listening interface (default: "127.0.0.1")
--metrics.port value Metrics HTTP server listening port (default: 6060)
--experimental.history.v3 (also known as Erigon3) Not recommended yet: Can't change this flag after node creation. New DB and Snapshots format of history allows: parallel blocks execution, get state as of given transaction without executing whole block.
--identity value Custom node name
--clique.checkpoint value number of blocks after which to save the vote snapshot to the database (default: 10)
--clique.snapshots value number of recent vote snapshots to keep in memory (default: 1024)
--clique.signatures value number of recent block signatures to keep in memory (default: 16384)
--clique.datadir value a path to clique db folder [<invalid Value>]
--watch-the-burn Enable WatchTheBurn stage to keep track of ETH issuance
--mine Enable mining
--proposer.disable Disables PoS proposer
--miner.notify value Comma separated HTTP URL list to notify of new work packages
--miner.gaslimit value Target gas limit for mined blocks (default: 30000000)
--miner.etherbase value Public address for block mining rewards (default: "0")
--miner.extradata value Block extra data set by the miner (default = client version)
--miner.noverify Disable remote sealing verification
--miner.sigfile value Private key to sign blocks with
--sentry.api.addr value comma separated sentry addresses '<host>:<port>,<host>:<port>'
--sentry.log-peer-info Log detailed peer info when a peer connects or disconnects. Enable to integrate with observer.
--downloader.api.addr value downloader address '<host>:<port>'
--no-downloader to disable downloader component
--downloader.verify verify snapshots on startup. it will not report founded problems but just re-download broken pieces
--healthcheck Enable grpc health check
--bor.heimdall value URL of Heimdall service (default: "http://localhost:1317")
--bor.withoutheimdall Run without Heimdall service (for testing purpose)
--ethstats value Reporting URL of a ethstats service (nodename:secret@host:port)
--override.terminaltotaldifficulty value Manually specify TerminalTotalDifficulty, overriding the bundled setting (default: <nil>) [<invalid Value>]
--override.mergeNetsplitBlock value Manually specify FORK_NEXT_VALUE (see EIP-3675), overriding the bundled setting (default: <nil>) [<invalid Value>]
--config value Sets erigon flags from YAML/TOML file
--verbosity value Logging verbosity: 0=silent, 1=error, 2=warn, 3=info, 4=debug, 5=detail (default: 3)
--log.json Format logs with JSON
--pprof Enable the pprof HTTP server
--pprof.addr value pprof HTTP server listening interface (default: "127.0.0.1")
--pprof.port value pprof HTTP server listening port (default: 6060)
--pprof.cpuprofile value Write CPU profile to the given file
--trace value Write execution trace to the given file
--help, -h show help
--version, -v print the version
+ return
aws-cli/2.8.9 Python/3.9.11 Linux/5.15.0-1019-aws exe/x86_64.ubuntu.20 prompt/off
+ return
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387630
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387632
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387633
+ prepare_zfs_datasets
+ set -euxo pipefail
+ zfs create -o mountpoint=/erigon/data tank/erigon_data
cannot create 'tank/erigon_data': dataset already exists
+ true
+ zfs create -o mountpoint=/erigon/data/bsc tank/erigon_data/bsc
cannot create 'tank/erigon_data/bsc': dataset already exists
+ true
+ download_snapshots
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/snapshots
+ download_nodes
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/nodes
+ download_parlia
+ set -euxo pipefail
+ safe_wait
+ zfs list tank/erigon_data/bsc/parlia
+ BACKGROUND_PIDS=($(jobs -p))
+ download_database_file
+ set -euxo pipefail
+ zfs list tank/erigon_data/bsc/chaindata
++ jobs -p
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387943
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/nodes 5.57M 2.17T 5.57M /erigon/data/bsc/nodes
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/parlia 116K 2.17T 116K /erigon/data/bsc/parlia
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/snapshots 599G 2.17T 599G /erigon/data/bsc/snapshots
+ aws s3 sync --request-payer requester s3://public-blockchain-snapshots/bsc/erigon-nodes-folder-latest/ /erigon/data/bsc/nodes/
+ mkdir -p /erigon/data/bsc/snapshots/
+ aws s3 cp --request-payer requester s3://public-blockchain-snapshots/bsc/parlia-db-latest.mdbx.zstd -
+ pzstd -d -o /erigon/data/bsc/parlia/mdbx.dat
NAME USED AVAIL REFER MOUNTPOINT
tank/erigon_data/bsc/chaindata 560G 2.17T 560G /erigon/data/bsc/chaindata
+ return
+ aws s3 sync --request-payer requester s3://public-blockchain-snapshots/bsc/erigon-snapshots-folder-latest/ /erigon/data/bsc/snapshots/
pzstd: /erigon/data/bsc/parlia/mdbx.dat already exists; do you wish to overwrite (y/n) ? pzstd: -: Not overwritten.
download failed: s3://public-blockchain-snapshots/bsc/parlia-db-latest.mdbx.zstd to - [Errno 32] Broken pipe
+ true
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387944
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387945
+ for PID in "${BACKGROUND_PIDS[@]}"
+ wait -f 1387947
+ prepare_erigon
+ set -euxo pipefail
+ useradd erigon
useradd: user 'erigon' already exists
+ true
+ chown -R erigon:erigon /erigon/data/
+ systemctl stop erigon-bsc
+ echo '[Unit]
Description=Erigon BSC daemon
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=1
User=erigon
ExecStart=/erigon/start_erigon_service.sh
[Install]
WantedBy=multi-user.target
'
+ echo '#!/bin/bash'
+ [[ '' == \1 ]]
+ echo -n 'exec erigon --chain bsc --snapshots=true --db.pagesize=16k --datadir=/erigon/data/bsc --txpool.disable'
+ [[ '' == \1 ]]
+ chmod +x /erigon/start_erigon_service.sh
+ systemctl daemon-reload
+ systemctl enable erigon-bsc
+ run_erigon
+ set -euxo pipefail
+ systemctl start erigon-bsc
+ add_create_snapshot_script
+ set -euxo pipefail
+ cat
+ chmod 0744 /erigon/create-bsc-snapshot-and-shutdown.sh
+ chown root:root /erigon/create-bsc-snapshot-and-shutdown.sh
+ echo 'create-bsc-snapshot-and-shutdown /erigon/create-bsc-snapshot-and-shutdown.sh uid=root erigon'
root@ip-172-31-5-127:/bsc-archive-snapshot#
from bsc-archive-snapshot.
allada
commented on June 11, 2024
You need to delete the tank/erigon_data/bsc/chaindata dataset
The script as written right now will not attempt to re-download chaindata file if the zfs dataset does not exist.
You can do this by running: zfs destroy tank/erigon_data/bsc/chaindata
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
okey, got it. let me stop the current download and delete, and restart, will update the status once finished
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
the download has been finished, then I run ./build_archive_node.sh again.
root@ip-172-31-5-127:/# df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.6G 6.1G 1.5G 81% /
devtmpfs 62G 0 62G 0% /dev
tmpfs 62G 0 62G 0% /dev/shm
tmpfs 13G 940K 13G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 62G 0 62G 0% /sys/fs/cgroup
/dev/nvme0n1p15 105M 5.2M 100M 5% /boot/efi
/dev/loop0 26M 26M 0 100% /snap/amazon-ssm-agent/5656
/dev/loop1 56M 56M 0 100% /snap/core18/2566
/dev/loop2 68M 68M 0 100% /snap/lxd/22753
/dev/loop3 64M 64M 0 100% /snap/core20/1623
/dev/loop4 48M 48M 0 100% /snap/snapd/16778
/dev/loop5 100M 100M 0 100% /snap/go/9981
tank/erigon_data 719G 128K 719G 1% /erigon/data
tank/erigon_data/bsc 719G 256K 719G 1% /erigon/data/bsc
tank/erigon_data/bsc/snapshots 1.3T 599G 719G 46% /erigon/data/bsc/snapshots
tank/erigon_data/bsc/parlia 719G 128K 719G 1% /erigon/data/bsc/parlia
tank/erigon_data/bsc/nodes 719G 5.5M 719G 1% /erigon/data/bsc/nodes
tmpfs 13G 0 13G 0% /run/user/1000
tank/erigon_data/bsc/chaindata 2.8T 2.1T 719G 75% /erigon/data/bsc/chaindata
the log of the service erigon-bsc:
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22658094-22658670 links=577 (22658094-22658670)} => 5c4e040c2132a7f1064f933515697c1a0a32edbe060d61c63597a58ac10c22be, anchorQueue.idx=3, next retry in 2.020729041s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22659822-22661742 links=1921 (22659822-22661742)} => 8e9545a1c73905189d5b8f99b8a2067983b0fdc5a159c1182e7ea074cab5551b, anchorQueue.idx=7, next retry in 2.020729041s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22663086-22663278 links=193 (22663086-22663278)} => c2508b07a54940a7ff146781a30d739b6b124b973a3ccd8e1b02d98d6598d893, anchorQueue.idx=1, next retry in 1.449934144s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22664046-22667886 links=3841 (22664046-22667886)} => 5dfbd61beda9b98ff6cc56d0746656d1b863c8c8a056c9b6b2063575025260f0, anchorQueue.idx=4, next retry in 2.020729041s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22668846-22672494 links=3649 (22668846-22672494)} => b96de5e780e81172fecbd8bed5845888727c27817dd56e30ae76646b608b85cb, anchorQueue.idx=5, next retry in 2.020729041s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22672878-22686318 links=13441 (22672878-22686318)} => d118340622325071c4d2b6cd45d896c5f14d3f296c68f22884b3ec87c072344a, anchorQueue.idx=6, next retry in 2.020729041s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22686702-22766190 links=79489 (22686702-22766190)} => cc07908bcb39db9b405e68dc4629af375d00e1c1b02a32619dd4ebc3fa8d297e, anchorQueue.idx=0, next retry in 1.332152192s
Nov 07 06:35:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:06.175] {22766574-22843211 links=76639 (22766574-22843210-22843211)} => a2147f1a57d57090001f47e2041262df4b993f652c3a938c35b2872006edc62b, anchorQueue.idx=2, next retry in 1.332152192s
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.170] Req/resp stats req=41 reqMin=22657325 reqMax=22766573 skel=15 skelMin=22658670 skelMax=22953390 resp=67 respMin=22657518 respMax=22843218 dups=859
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.176] [Downloader] Queue sizes anchors=1 links=185702 persisted=65536
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.176] {22657518-22843218 links=185702 (22657518-22843210-22843218)} => 9935247d6b9b33a6a1ca4486754c07e971e8398155be1f3634833b6306e04452, anchorQueue.idx=0, next retry in 4.813953301s
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.635] [parlia] snapshots build, gather headers block=22600000
Nov 07 06:35:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|06:35:26.885] [parlia] snapshots build, gather headers block=22500000
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Yep, now it needs to sync back up. This part may take many hours to a couple days.
from bsc-archive-snapshot.
allada
commented on June 11, 2024
You should eventually see something like:
[INFO] [11-07|06:37:44.064] [7/16 Execution] Blocks execution from=22843214 to=22843262
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
okey, I will keep it running there. So when the node keeps sync up with the latest block no, I would be able to get all the receipt and trace data of all blocks, is this correct?
e.g. eth_getTransactionReceipt, debug_traceBlockByNumber, debug_traceTransaction
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Yes.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
that's cool. thx very much!
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
got the following output, what does this mean? is this expected?
Nov 07 07:24:46 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:24:46.170] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
Nov 07 07:25:02 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:02.090] [p2p] GoodPeers eth66=23
Nov 07 07:25:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:06.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 07:25:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:06.170] Req/resp stats req=0 reqMin=0 reqMax=0 skel=19 skelMin=22658670 skelMax=22953390 resp=11 respMin=0 respMax=0 dups=0
Nov 07 07:25:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:06.170] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
Nov 07 07:25:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:26.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 07:25:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:26.170] Req/resp stats req=0 reqMin=0 reqMax=0 skel=18 skelMin=22658670 skelMax=22953390 resp=12 respMin=0 respMax=0 dups=0
Nov 07 07:25:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:26.170] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
Nov 07 07:25:46 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:46.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 07:25:46 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:46.170] Req/resp stats req=0 reqMin=0 reqMax=0 skel=18 skelMin=22658670 skelMax=22953390 resp=16 respMin=0 respMax=0 dups=0
Nov 07 07:25:46 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:25:46.170] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
Nov 07 07:26:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:06.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 07:26:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:06.170] Req/resp stats req=0 reqMin=0 reqMax=0 skel=18 skelMin=22658670 skelMax=22953390 resp=10 respMin=0 respMax=0 dups=0
Nov 07 07:26:06 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:06.171] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
Nov 07 07:26:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:26.170] [2/16 Headers] No block headers to write in this log period block number=22657134 blk/second=0.000
Nov 07 07:26:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:26.170] Req/resp stats req=0 reqMin=0 reqMax=0 skel=17 skelMin=22658670 skelMax=22953390 resp=17 respMin=0 respMax=0 dups=0
Nov 07 07:26:26 ip-172-31-5-127 start_erigon_service.sh[3664412]: [INFO] [11-07|07:26:26.170] [Downloader] Queue sizes anchors=0 links=0 persisted=65536
from bsc-archive-snapshot.
allada
commented on June 11, 2024
It can take a few hours to get going. But if you get stuck, delete the files in the snapshots folder then re-download it.
Those error messages are a known issue that has been happening lately that the erigon team is aware of.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
yes, it's stuck, I am redownloading the snapshots now...
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Hold up, i tried to create a node from this version and there's an issue ... I'm uploading a new one, should be done in a couple hours.
…-------- Original Message --------
On Nov 7, 2022, 8:23 PM, bdonglobal wrote:
yes, it's stuck, I am redownloading the snapshots now...
—
Reply to this email directly, [view it on GitHub](#38 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAN7CIVHF2IZMRCCDETIRGDWHG2T3ANCNFSM6AAAAAARPP7FKU).
You are receiving this because you commented.Message ID: ***@***.***>
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
okey
from bsc-archive-snapshot.
allada
commented on June 11, 2024
It is still uploading and will likely take another ~2 hours and I'm probably going to sleep soon. When the snapshot is done it'll be located here:
s3://public-blockchain-snapshots/snap.zfs.zstd
You'll need to delete the existing datasets using something like: zfs destroy tank/erigon_data -R -f
The command to download and load it up will be something like:
s3pcp --requester-pays s3://public-blockchain-snapshots/snap.zfs.zstd | pv | pzstd -p $(nproc) -q -d -f | zfs recv tank/erigon_data
This should create a bunch of zfs sub-datasets too. I'm not sure if they will mount properly, but if they don't, mount them so they look like this:
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 2.91T 401G 96K none
tank/erigon_data 2.89T 401G 96K /erigon/data
tank/erigon_data/bsc 2.89T 401G 10.9M /erigon/data/bsc
tank/erigon_data/bsc/chaindata 2.34T 401G 2.02T /erigon/data/bsc/chaindata
tank/erigon_data/bsc/nodes 10.1M 401G 5.31M /erigon/data/bsc/nodes
tank/erigon_data/bsc/parlia 1.59M 401G 1.45M /erigon/data/bsc/parlia
tank/erigon_data/bsc/snapshots 562G 401G 562G /erigon/data/bsc/snapshots
(only the ones that start with tank/erigon_data matter).
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
okey, I run the following cmds and it is now downloading...
zfs destroy tank/erigon_data -R -f
s3pcp --requester-pays s3://public-blockchain-snapshots/snap.zfs.zstd | pv | pzstd -p $(nproc) -q -d -f | zfs recv tank/erigon_data
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
for the mount, I run the following cmds:
zfs set mountpoint=/erigon/data tank/erigon_data
zfs set mountpoint=/erigon/data/bsc tank/erigon_data/bsc
zfs set mountpoint=/erigon/data/bsc/chaindata tank/erigon_data/bsc/chaindata
zfs set mountpoint=/erigon/data/bsc/nodes tank/erigon_data/bsc/nodes
zfs set mountpoint=/erigon/data/bsc/parlia tank/erigon_data/bsc/parlia
zfs set mountpoint=/erigon/data/bsc/snapshots tank/erigon_data/bsc/snapshots
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
then I use systemctl restart erigon-bsc to restart the erigon:
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.349] new subscription to logs established
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.349] rpc filters: subscribing to Erigon events
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.349] new subscription to newHeaders established
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.349] Reading JWT secret path=/erigon/data/bsc/jwt.hex
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.349] HTTP endpoint opened for Engine API url=localhost:8551 ws=true ws.compression=true
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.350] HTTP endpoint opened url=localhost:8545 ws=false ws.compression=true grpc=false
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.364] [1/16 Snapshots] Fetching torrent files metadata
Nov 08 08:09:07 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:07.370] Started P2P networking version=66 self=enode://482e40ff75dbebdffa39af96b4bcf93e3c0bdec8cd3636a69587114101c1947748625aca4b6b53021640302516a5edd21121a2bb97245e2d9ad6250121747023@127.0.0.1:30303 name=erigon/v2.28.1-stable-9711af6d/linux-amd64/go1.18.7
Nov 08 08:09:11 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:11.249] [Snapshots] Blocks Stat blocks=22775k indices=22775k alloc=2.4GB sys=2.9GB
Nov 08 08:09:11 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:11.251] [2/16 Headers] Waiting for headers... from=22865395
Nov 08 08:09:31 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:31.252] [2/16 Headers] Wrote block headers number=22873450 blk/second=402.750 alloc=2.7GB sys=2.9GB
Nov 08 08:09:51 ip-172-31-5-127 start_erigon_service.sh[144521]: [INFO] [11-08|08:09:51.252] [2/16 Headers] No block headers to write in this log period block number=22873450 blk/second=0.000
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
when I use the debug* commands, it says "does not exist/is not available", then how to enable this?
root@ip-172-31-5-127:/# curl -X POST -H "Content-Type:application/json" --data '{"jsonrpc": "2.0", "id": 56, "method": "debug_traceBlockByNumber", "params": ["0x1460bc5",{"tracer": "callTracer","disableStorage":"true","disableMemory":"true","disableStack":"true","timeout":"600s"}]}' http://127.0.0.1:8545
{"jsonrpc":"2.0","id":56,"error":{"code":-32601,"message":"the method debug_traceBlockByNumber does not exist/is not available"}}
root@ip-172-31-5-127:/#
root@ip-172-31-5-127:/#
root@ip-172-31-5-127:/#
root@ip-172-31-5-127:/# curl -X POST -H "Content-Type:application/json" --data '{"jsonrpc": "2.0", "id": 56, "method": "debug_traceTransaction", "params": ["0xed42802e8a4d1ea5e10a09890d6857ada9aff47fa3f7e08497b4f2a5246e252d"]}' http://127.0.0.1:8545
{"jsonrpc":"2.0","id":56,"error":{"code":-32601,"message":"the method debug_traceTransaction does not exist/is not available"}}
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
it works well after adding "--http.api "eth,net,web3,admin,debug,txpool"
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
I am trying to move the data from ephemeral to ebs(newly created 4T SSD), but the copy failed eventually due to no space, then I checked the disk usage, and found the following result. Why the total is 2.1T, and the mdbx.dat is 4.7T? how much size disk totally you recommend if switching to SSD ebs?
root@ip-172-31-5-127:/erigon/data/bsc/chaindata# ls -lh
total 2.1T
-rw-r--r-- 1 erigon erigon 4.7T Nov 8 09:59 mdbx.dat
-rw-r--r-- 1 erigon erigon 0 Nov 8 10:00 mdbx.lck
from bsc-archive-snapshot.
allada
commented on June 11, 2024
When you ran the zfs recv command it imported it as a snapshot, you need to delete the snapshots to free up the space.
zfs destroy tank/erigon_data@snap
zfs destroy tank/erigon_data/bsc@snap
zfs destroy tank/erigon_data/bsc/chaindata@snap
zfs destroy tank/erigon_data/bsc/snapshots@snap
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
So it means I should run the above commands firstly to remove the snapshots, then cp them to SSD? otherwise the snapshots are also being copied to SSD?
from bsc-archive-snapshot.
allada
commented on June 11, 2024
I miss read your original post...
The filesystem lies about the actual size of the file. On disk it really is ~4.7TB, but it's actually much larger than that uncompressed.
It looks like you are probably trying to copy it onto a disk that is not compressed. You can get the real size of the file using:
du --apparent-size
You are likely going to need ~6-7TB if you want it on a filesystem uncompressed.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
got it, ~6-7TB is too huge size
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
the reason of using EBS is, I don't know how to extend the size of the current instance's ephemeral storage. I asked AWS, they said this kind of resouce cannot be extended.
from bsc-archive-snapshot.
allada
commented on June 11, 2024
Correct, you need to create a new instance in order to get larger ephemeral storage. This kind of support goes beyond the scope of this project. Feel free to read deep into how the build_archive_node.sh script works and what it does, it may give you some ideas on how to quickly transfer the files from server to server, or server to S3 to server.
from bsc-archive-snapshot.
bdonglobal
commented on June 11, 2024
got it. thx very much. I have 700g+ free space for now. will prepare a new one for later usage
from bsc-archive-snapshot.
Related Issues (20)
- how login account HOT 1
- AWS fatal error: Unable to locate credentials HOT 10
- Problem when calling the ListObjectsV2 operation: Access Denied HOT 3
- The whole snapshot only 500GB? HOT 3
- ETH vs BSC Archive Nodes` HOT 1
- fatal error: An error occurred (404) when calling the HeadObject operation: Key "bsc/erigon-latest.tar.zstd" does not exist HOT 1
- Download snapshots failed HOT 7
- How to store 30TB in AWS EBS? HOT 4
- Sync slow HOT 8
- fail to start bsc archive with new snapshot HOT 5
- Empty file after downloading erigon-bsc-archive HOT 2
- Current version failure to download/run... HOT 2
- The script stopped working HOT 3
- Script finishes with no errors at 568GB. HOT 9
- getting block not found for older blocks HOT 1
- Always about 8 minutes behind the latest block HOT 5
- S3 folder structure possibly wrong HOT 4
- On (probable) discontinuation of erigon support for BSC HOT 4
- Slow synchronization HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bsc-archive-snapshot.