Allow Importing of Historical Blobs about prysm

[nisdas]

@kaber2 OffchainLabs/nitro#2231 (comment) Taking the conversation here. Backfilling blocks and blobs are separate concerns. Every full node should have all the historical blocks, but there is no guarantee that a full node will have historical blobs. If we do request blobs beyond the retention period(4096 epochs) then it is pretty likely that an average peer in the network will not hold it because they have already pruned it.

Purpose of this issue is to allow anyone who wants to hold all the historical blobs to simply provide a historical blob rpc provider, and prysm will request it from them and fill it internally on its blob directory.

[ZYS980327]

@nisdas So how should we fix this data, I restarted a prysm and added the flag --blob-retention-epochs=999999999, don't know if it's useful, but still see some logs printed about fetch batches or slot failures, where the peer doesn't have these data

[nisdas]

@ZYS980327 Like i mentioned earlier, you can use the archival rpc provider to sync the missing blobs till your arbitrum node is at its tip and then switch over to using your local prysm node.

[ZYS980327]

@nisdas I don't know how to reboot the nitro node, every reboot is resynced at the very beginning of the snapshot, I redeployed the Ethereum nodes geth and prysm today, hope it helps

[nisdas]

I am pretty sure this is your issue:
OffchainLabs/nitro#2231 (comment)

If you remove that flag it should be fine. You should never be using that flag for any long lived process with persistent state as it will remove all the data.

[ZYS980327]

@nisdas Yes, I removed this flag today, but it still doesn't work, nitro is which directory was retrieved

[nisdas]

When you stop the process, can you confirm the directory exists ?

[ZYS980327]

@nisdas I'm using docker kill [arg], the directory should be in the docker-related directory, it always exists, it's all hash value directory, I just don't know which one

[kaber2]

@kaber2 OffchainLabs/nitro#2231 (comment) Taking the conversation here. Backfilling blocks and blobs are separate concerns. Every full node should have all the historical blocks, but there is no guarantee that a full node will have historical blobs. If we do request blobs beyond the retention period(4096 epochs) then it is pretty likely that an average peer in the network will not hold it because they have already pruned it.

Purpose of this issue is to allow anyone who wants to hold all the historical blobs to simply provide a historical blob rpc provider, and prysm will request it from them and fill it internally on its blob directory.

According to the release announcement (https://github.com/prysmaticlabs/prysm/releases/tag/v5.0.0), this flag does support backfilling blobs. After adding the aforementioned flag, my node did sync a large number of blobs from the network and is now at:

$ ls |wc -l
61361

According to the logs, backfilling is complete. So just to be 100% clear, you are saying that I still don't hold all blobs?

[nisdas]

You will only backfill blobs till the retention period(4096 epochs). There is no guarantee that the p2p network will be able to supply older blobs beyond that, which is why we do not sync older blobs.

[kaber2]

I see. Thank you for the explanation. I'll try to find a suitable RPC provider then.