Sync slow about bsc-archive-snapshot

[allada]

It looks like you need to run it on better hardware. More memory and faster disk latency. SSDs can barely keep up, so use NVMSs instead.

[naviocean]

@allada I'm using SSD with 7500 IOPS, 16. Do I need to update?

[allada]

Probably... Check your disk IO, it's probably maxed reading. Don't use EBS or other non ephemeral storage, the latency is too high.

You can also try using EXT4 instead of zfs, this sometimes gives better performance, but you should be getting closer to 8 to 14 blocks per second.

[naviocean]

@allada I've terminated all and try again. Could you please confirm the size of data and snapshots are correct?

And how long to pass the snapshots indexing progress? the size of /temp is increasing.

erigon_1  | [INFO] [11-13|10:36:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h1m20s alloc=4.8GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h1m40s alloc=6.2GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:38.150] [p2p] GoodPeers                          eth66=90
erigon_1  | [INFO] [11-13|10:36:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m0s alloc=7.6GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:36:52.773] [index] write                            file=v1-012500-013000-transactions.idx
erigon_1  | [INFO] [11-13|10:37:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m20s alloc=8.3GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:37:31.190] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h2m40s alloc=9.1GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:37:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m0s alloc=5.7GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:11.198] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m20s alloc=7.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h3m40s alloc=9.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:38:38.150] [p2p] GoodPeers                          eth66=90
erigon_1  | [INFO] [11-13|10:38:51.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m0s alloc=6.9GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:39:11.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m20s alloc=9.0GB sys=11.9GB
erigon_1  | [INFO] [11-13|10:39:31.188] [1/16 Snapshots] Indexing                progress="v1-012500-013000-transactions.seg=50%, v1-013000-013500-transactions.seg=50%, v1-013500-014000-transactions.seg=50%" total-indexing-time=5h4m40s alloc=5.9GB sys=11.9GB

I'm using the docker-compose for running Erigon

version: '3.9'

x-logging: &logging
  logging:
    driver: "json-file"
    options:
      max-size: "2m"
      max-file: "10"

services:
  erigon:
    image: thorax/erigon:v2.29.0
    command: |
      erigon --chain bsc
      --snapshots=true
      --db.pagesize=16k
      --datadir=/home/erigon/.local/share/erigon
      --metrics --metrics.addr=0.0.0.0 --metrics.port=6060
      --private.api.addr=0.0.0.0:9090
      --pprof --pprof.addr=0.0.0.0 --pprof.port=6061
      --torrent.port=42069 --torrent.upload.rate=15mb --torrent.download.rate=1024mb
      --authrpc.jwtsecret=/home/erigon/.local/share/erigon/jwt.hex
      --txpool.disable
    volumes:
      - /erigon/data/mainnet:/home/erigon/.local/share/erigon
    ports:
      - 30303:30303/tcp
      - 30303:30303/udp
      - 30304:30304/tcp
      - 30304:30304/udp
      - 9090:9090
      - 8551:8551
    restart: unless-stopped
    mem_swappiness: 0
    <<: *logging

[allada]

Indexing usually takes a few hours. If you change the mtime of the .idx files it may allow you to skip this stage. The sizes look appropriate. The temp folder is used to store ETL data.

[naviocean]

@allada I faced a slow execution again

erigon_1      | [INFO] [11-13|17:51:24.477] [7/16 Execution] Executed blocks         number=22899922 blk/s=0.2 tx/s=25.7 Mgas/s=4.1 gasState=0.01 batch=15.2MB alloc=2.3GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:51:41.857] [7/16 Execution] Executed blocks         number=22899931 blk/s=0.5 tx/s=52.9 Mgas/s=6.5 gasState=0.01 batch=15.5MB alloc=2.3GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:01.290] [7/16 Execution] Executed blocks         number=22899938 blk/s=0.4 tx/s=69.9 Mgas/s=7.3 gasState=0.01 batch=15.8MB alloc=2.4GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:38.024] [7/16 Execution] Executed blocks         number=22899943 blk/s=0.1 tx/s=16.3 Mgas/s=3.6 gasState=0.01 batch=16.2MB alloc=2.5GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:42.239] [7/16 Execution] Executed blocks         number=22899946 blk/s=0.7 tx/s=106.5 Mgas/s=10.0 gasState=0.01 batch=16.3MB alloc=2.5GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:52:46.913] [p2p] GoodPeers                          eth66=24
erigon_1      | [INFO] [11-13|17:53:01.571] [7/16 Execution] Executed blocks         number=22899954 blk/s=0.4 tx/s=62.1 Mgas/s=6.6 gasState=0.01 batch=16.6MB alloc=2.2GB sys=3.8GB
erigon_1      | [INFO] [11-13|17:53:50.678] [7/16 Execution] Executed blocks         number=22899964 blk/s=0.2 tx/s=29.2 Mgas/s=4.4 gasState=0.01 batch=17.1MB alloc=2.4GB sys=3.8GB

I've updated my SSD to 16000 IOPS and Max throughput 500, But execution speed is too low. Do you have any suggestion or tools to check the main of problem? I'm using Azure service instead of AWS

[allada]

You need to use an ephemeral disk to capitalize of the low latency.

You can also try creating many disks and raid them together (you'll need to then re-copy the data on the raid). This might help some.

[naviocean]

@allada thank you so much. I've set raid for 4 disks and the issue is solved. The speed of execution now is reached to 20. :)