Current unstable failing forkchoiceUpdated requests about nimbus-eth2

[jakubgs]

Originally this issue was discovered because some Waku fleets use the `` host for RLN relay:

# Waku rln-relay parameters
nim_waku_rln_relay_dynamic: true
nim_waku_rln_relay_eth_contract_address: '0xF471d71E9b1455bBF4b85d475afb9BB0954A29c4'
nim_waku_rln_relay_eth_client_address: 'ws://linux-01.ih-eu-mda1.nimbus.sepolia.wg:9557'
nim_waku_rln_relay_tree_path: '/data/rln_relay_tree'

[- infra-waku#c96047f3 - node-db: disable replication explicitly](https://github.com/status-im/infra-waku/blob/e97cb8e8933ec12ac05bb87fa260de9066ece041/ansible/group_vars/node.yml#L58-L62)

Which manifested as following error:

Mounting protocols failed
    topics="wakunode main"
    tid=1 file=wakunode2.nim:89
    error="failed to mount waku RLN relay protocol: failed to mount WakuRlnRelay: exception in new WakuRlnRelay: Membership index is not set"

Which started today at 01:00 AM:

[jakubgs]

I have reverted unstable-01 to the commit before 017f9f11, which is c815e71a and that has resolved the issue, the errors are gone:

[email protected]:~ % j build-beacon-node-sepolia-unstable-01.service | grep Finished
Jan 16 09:33:14 linux-01.ih-eu-mda1.nimbus.sepolia systemd[1]: Finished Build beacon-node-sepolia-unstable-01.

[email protected]:~ % zgrep 'Failed EL Request.*forkchoiceUpdated' /var/log/service/beacon-node-sepolia-unstable-01/service.* | tail -n1  
/var/log/service/beacon-node-sepolia-unstable-01/service.20240116_1000.log.gz:{"lvl":"DBG","ts":"2024-01-16 09:33:08.563+00:00","msg":"Failed EL Request","topics":"elmon","requestName":"forkchoiceUpdated","statusCode":0,"err":"missing or invalid response id"}

And the respective Geth node is syncing:

[email protected]:~ % for dir in /docker/geth-*; do $dir/rpc.sh eth_syncing | jq -c; done
{"jsonrpc":"2.0","id":1,"result":{"currentBlock":"0x22edb4","healedBytecodeBytes":"0x0","healedBytecodes":"0x0","healedTrienodeBytes":"0x0","healedTrienodes":"0x0","healingBytecode":"0x0","healingTrienodes":"0x0","highestBlock":"0x4dc084","startingBlock":"0x0","syncedAccountBytes":"0x0","syncedAccounts":"0x0","syncedBytecodeBytes":"0x0","syncedBytecodes":"0x0","syncedStorage":"0x0","syncedStorageBytes":"0x0"}}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}

Unlike the others still using a broken beacon node.

[jakubgs]

I have changed the layout of Sepolia fleet to use stable, testing, and libp2p branches and not just unstable:

• infra-nimbus#548f0f48 - nimbus.sepolia: change layout to use all main branches

This way the first node that Waku fleets depend on for RLN will not easily break.

[jakubgs]

Resolved by:

• #5759