dappnode / dappnodepackage-prysm-prater

[Prater testnet] Wrapper for the Prysm Eth2 client, beacon and validator binaries

Prysm DAppNode package (prater config)

Prysm prater ETH2.0 Beacon chain + validator

Validate with prysm: a Go implementation of the Ethereum 2.0 Serenity protocol and open source project created by Prysmatic Labs. Beacon node which powers the beacon chain at the core of Ethereum 2.0

Grafana dashboard thanks to the amazing work of metanull-operator

Updated	Champion/s
✔️	@tropicar

Development

1. Select a Prysm branch, current work progress use to be tracked at develop branch. Set the branch env in the docker-compose.dev.yml

2. Build Prysm with the branch defined with

npx @dappnode/dappnodesdk build --compose_file_name=docker-compose.dev.yml

dappnodepackage-prysm-prater's Issues

Prysm Prater allowlist headers

Detailed in dappnode/DAppNode#461
1/4 of dappnode/DAppNode#468

Add headers to the RPC requests defining the CL so the EL allowlist dappnode/DAppNode#462 and dappnode/DAppNode#463 can work.

Implement JSON Web Token

The upcoming new release of Prysm will entail a JSON Web Token in the UI that must be sent to the dappmanager so the user must use it in the first login. The data sent must contain the URL to be used in the first login so the dappmanager recognizes it and render it as a URL.

TODO

• Add a beauty explanation in the getting-started.yml, dappnode_package.json (in the description) or in both about how and when to use this URL
• Build and send the URL with the JWT to the dappmanager in the entrypoint.sh taking as reference the code from https://github.com/dappnode/DAppNodePackage-prysm-prater/tree/pablo/test-ui

Consider not persisting public_keys.txt file in a docker volume

Related to #36

Consider not persisting public_keys.txt file in a docker volume so every time the container restarts it will have to fetch the public keys from the web3signer without doing any assupmtions

Implement build from source

Determine a way to build Prysm from source taking as a reference https://github.com/dappnode/DAppNodePackage-prysm-prater/tree/pablo/test-ui and create a Dockerfile_test for the sources. This will allow testing Prysm releases without having to explicit way for the release comming up

Todo

• build validator
• build beacon chain

Prysm Prater does not designate its activities BeaconChain and Validator as Prater So Grafana combines the data from both in the docker dashboard

Copy and repost of dappnode/DAppNodePackage-DMS#26 fixing this issue will fix dappnode/DAppNodePackage-DMS#26 as welll.

Describe the bug
The grafana docker dashboard doesn't appear to show Prysm Prater Beacon-chain and Validator. when running a mainnet prysm package at the same time. they appear to be combined into one service for each sub-service, both mainnet and prater beaconchain, and then both mainnet and prater validators.

To Reproduce
Install DMS, DAppNodeExporter, Prysm Prater, GoerliEth, geth light, and prysm.
let everything sync and go to the dms dashboard for docker and you will see what seems to be an outsize amount of resources dedicated to beacon-chain, and validator. shut off the prysm and youll se a huge drop in resource usage but not entirely. it continues to run at prater speed.

Expected behavior
I would expect to see a beacon chain metric, a prater beacon chain metric, a prysm validator metric, and a prysm prater validator metric

Screenshots
with both prater and mainnet running

shut off mainnet prysm

dashboard after shutting off prysm mainnet

DAppNode version:
@mgarciate sees the same, and i see it on all 5 of my dappnodes

Praters metrics does not work in DMS package

Describe the bug

Grafana (DMS + node exporter package) does not show any metrics of the prysm-prater package.

To Reproduce

Steps to reproduce the behaviour:

1. Install DMS, node exporter and prysm prater packages
2. Go to grafana and select the prater dashboard
3. You will not see any metrics working

Expected behavior

You should be able to see the different dashboards working.
If you go to http://prometheus.dms.dappnode:9090/targets (connected to your dappnode via VPN). You will see this error:

Get "http://beacon-chain.prater.dappnode:8080/metrics": dial tcp 172.33.0.15:8080: connect: connection refused

We can see the name is wrong, it should be http://beacon-chain.prysm-prater.dappnode:8080/metrics

I released the file where metrics are set up in this repo is not correct, I replaced the wrong name prater for prysm-prater but anyway the errors persist.

Another interesting thing it's, if you go manually to http://beacon-chain.prysm-prater.dappnode:8080/metrics you can see the metrics. But if you go to grafana dashboard and try to edit some dashboard and use this service name, it does not work.

Screenshots

Additional context

This package is almost similar to prysm-pyrmont package where the metrics work.

Large and unusual CPU usage from validator process since upgrade to version 1.0.1

Describe the bug

Large and unusual CPU usage from validator process since upgrade to version 1.0.1.

To Reproduce

1. Install version 1.0.1
2. Sync beacon chain with no validator in the keystore
3. See large number of warning logs with message: [ WARN ] no public keys found.
4. See large CPU usage from validator.prysm-prater docker container

Expected behavior

Validator process should use little to no CPU.

Screenshots

DAppNode version:

Core DAppNode Packages versions

bind.dnp.dappnode.eth: 0.2.6
core.dnp.dappnode.eth: 0.2.52
dappmanager.dnp.dappnode.eth: 0.2.47, commit: 35c4a081
https.dnp.dappnode.eth: 0.1.3
ipfs.dnp.dappnode.eth: 0.2.15
vpn.dnp.dappnode.eth: 0.2.8, commit: f9a8743e
wifi.dnp.dappnode.eth: 0.2.8
wireguard.dnp.dappnode.eth: 0.1.1

System info

dockerComposeVersion: 1.25.5
dockerServerVersion: 20.10.2
dockerCliVersion: 20.10.2
os: debian
versionCodename: bullseye
architecture: amd64
kernel: 5.10.0-5-amd64
Disk usage: 52%

Additional context

Prysm Prater instance is only used as a beacon chain for SSV network validation, validator process is not being used. Validator process had to be turned off manually to regain wasted CPU resource (about 12%!)

Validator remote keymanage API does not work if beacon syncing

The validator remote keymanager API is not available while the beacon node is in status syncing. There is a check to the beacon node api to get the sync status and not move forward if true.

Keep an eye on the upstream to check when this issue get fixed

Track Prysm

Track the most important features in the road to Eth2 multiclient in DAppNode of the Prysm client.

• Implement web3signer support
• Implement validator remote keymanager API: #55
• Fee recipient: #73
• Checkpoint sync 🚀 prysmaticlabs/prysm#10387
• DAppNode package for Gnosis chain dappnode/DAppNodePackage-gnosis-beacon-chain-prysm#14
• DAppNode package for mainnet
• Implement JWT

Test Eth2 migration in network Prater

Test instructions

Having at least 2 validators in the legacy Prysm-prater (previous version to v1.0.0) and web3signer-prater already installed, manually update Prysm-prater to the latest version.

During container start-up, there should be some logs saying "found validators, starting migration". This process will export your validator keystores, walletpassword, and slashing protection data to the web3signer via the remote keymanager API.

If there is any error during this migration, the volume will be empty, ensuring no keystores are loaded in the Prysm-prater.

Ensure this process was successfully and the keystores are loaded in the web3signer

Test report

With an editable unique comment report any errors and feedback you may find necessary.

If no errors leave a checkmark

Testers

@pablomendezroyo ✔️
@3alpha
@tropicar
@mgarciate
@Pol-Lanski ✅

Prepare prysm prater for the merge

It's not a bug, this issue is a task in order to add the required configuration for the merge on Goerli network.

• Add flag to add the JWT file path

--jwt-secret=

• Add override terminal difficulty flag

This flag has been required on testnets, but possibly on mainet is not required. But anyway, we have it ready for adding it if it's required. In mainet possibly this value is added in the client code. That is the reason that is not required to add it as a flag.

--terminal-total-difficulty-override=$TTD_OVERRIDE

• Indicate API engine endpoint

Prysm client does have a new flag for API engine, instead, the value of the flag which the package has already set up --http-web3provider=$HTTP_WEB3PROVIDER that selects the endpoint 8545 of the execution client, has to be changed for the port 8551 during the Bellatrix upgrade.

Test Prysm + web3signer in network prater

Test instructions

You must have at least 2 Prater validators.

Install the web3signer prater and the client Prysm. You must get the hash from the Github latest release "dot".

In the client use the checkpoint sync feature to be able to start validating in minutes.

The web3signer will listen only to one client, you must select the client Prysm at config.

For a deeper test, you may find it useful to check out the grafana dashboards from the client, web3signer, and docker host.

Test report

With an editable unique comment report any errors and feedback you may find necessary.

Testers

• @pablomendezroyo ✔️
• @3alpha
• @tropicar ✔️
• @mgarciate

Web3 Failover String does not like the Old Failover String Format, and causes the beacon chain to crash

Describe the bug

Upgraded to Version: 1.0.5 (v2.1.2 upstream)

After update, the Prysm Prater beacon chain stopped synching

Noticed that it kept crashing shortly after saying it was going to try the failover provider.

On a hunch, deleted my old web3 failover string, which was living in the EXTRA_OPTS field. The old string looked like this:

--fallback-web3provider=https://eth-goerli.alchemyapi.io/v2/API-SECRET-GOES-HERE --fallback-web3provider=https://goerli.infura.io/v3/API-SECRET-GOES-HERE

After deleting that string, the beacon chain began to sync properly. When it finished syncing, the test validators began working properly, too.

DAppNode version:

Before filing a new topic, please provide the following information.
Core DAppNode Packages versions

bind.dnp.dappnode.eth: 0.2.6
core.dnp.dappnode.eth: 0.2.54
dappmanager.dnp.dappnode.eth: 0.2.49, commit: 14f56e8e
https.dnp.dappnode.eth: 0.1.3
ipfs.dnp.dappnode.eth: 0.2.16
wifi.dnp.dappnode.eth: 0.2.8
wireguard.dnp.dappnode.eth: 0.1.1

System info

dockerComposeVersion: 1.25.5
dockerServerVersion: 20.10.6
dockerCliVersion: 20.10.6
os: debian
versionCodename: bullseye
architecture: amd64
kernel: 5.10.0-8-amd64
Disk usage: 78%

Additional context

Add checkpoint sync

Add the checkpoint sync feature (as it is in lighthouse and teku) once its released in Prysm

Hardcoded the JWT using the entrypoint

Describe the bug
Prysm overrides the JWT and the validator does not work.

To Reproduce

Steps to reproduce the behaviour:

1. Install version 2.1.2 of prysm prater
2. Go docker exec -it DAppNodePackage-validator.prysm-prater.dnp.dappnode.eth /bin/sh
3. Go to /root/.eth3validators/auth-token and change the value of that token
4. Restart the container
5. Check the value, the value is the same after the change, which means the data in the volume persists over the operation are done in the dockerfile(where is hardcoded the jwt).

Expected behavior

That token must be the same after a restart.

Allow to change the JWT

Allow to change the JWT necessary to get into the UI on demand by the user

depends on prysmaticlabs/prysm-web-ui#214

web3signer-ui does not import keys

Some time ago I imported 3 keystores and everything was working as expected.
Today I checked my validators and upon accessing the UI page I was redirected to the empty Your Validator Accounts List screen.

Prysm Prater validator logs show:
time="2022-05-30 02:23:14" level=warning msg="No validating keys fetched. Trying again" prefix=validator

Attempt to re-import my keys end up in Import Completed popup with the following message:
Error: Failed to fetch

I tried to delete and reinstall the web3signer package - the same result.

To Reproduce

Expected behavior

Screenshots

DAppNode version:

Core DAppNode Packages versions
bind.dnp.dappnode.eth: 0.2.6
core.dnp.dappnode.eth: 0.2.54
dappmanager.dnp.dappnode.eth: 0.2.49, commit: 14f56e8e
https.dnp.dappnode.eth: 0.1.3
ipfs.dnp.dappnode.eth: 0.2.16
vpn.dnp.dappnode.eth: 0.2.8, commit: f9a8743e
wifi.dnp.dappnode.eth: 0.2.8
System info
dockerComposeVersion: 1.25.5
dockerServerVersion: 20.10.2
dockerCliVersion: 20.10.2
os: debian
versionCodename: bullseye
architecture: amd64
kernel: 5.10.0-1-amd64
Disk usage: 81%

• Package version: Version: 0.1.4 (22.5.0 upstream)
• OS: DAppNode Advance
• Browser: Edge

Additional context

web3signer status is 3/4 running and 1/4 restarting

Web3signer Prater logs:

1. UI

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/05/30 01:54:47 [notice] 1#1: using the "epoll" event method
2022/05/30 01:54:47 [notice] 1#1: nginx/1.21.6
2022/05/30 01:54:47 [notice] 1#1: built by gcc 10.3.1 20211027 (Alpine 10.3.1_git20211027) 
2022/05/30 01:54:47 [notice] 1#1: OS: Linux 5.10.0-1-amd64
2022/05/30 01:54:47 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/05/30 01:54:47 [notice] 1#1: start worker processes
2022/05/30 01:54:47 [notice] 1#1: start worker process 33
2022/05/30 01:54:47 [notice] 1#1: start worker process 34
2022/05/30 01:54:47 [notice] 1#1: start worker process 35
2022/05/30 01:54:47 [notice] 1#1: start worker process 36
2022/05/30 01:54:47 [notice] 1#1: start worker process 37
2022/05/30 01:54:47 [notice] 1#1: start worker process 38
2022/05/30 01:54:47 [notice] 1#1: start worker process 39
2022/05/30 01:54:47 [notice] 1#1: start worker process 40

2. web3signer

Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
[0.004s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /opt/web3signer/hs_err_pid7.log
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
[0.003s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# /opt/web3signer/hs_err_pid7.log
Setting up watches.  Beware: since -r was given, this may take a while!
Watches established.
[0.002s][warning][os,thread] Failed to start thread - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

Grafana doesn't show uptime for validator and beacon-chain

Infura backup doesn't work for Prysm Prater ("no service beacon-validator...")

@etherni thank you for the report!

I created a new ETH2 project on Infura and noticed a new connection string template:

The mainnet naming was a typo, you should always use prater endpoint. You can use the same RPC endpoint for the checkpoint sync and for the web3 fallback provider

This typo was fixed by #70

Originally posted by @pablomendezroyo in #60 (comment)

However, Infura backup seems not working - upon specifying Infura Prater endpoint I see "no service beacon-validator..." message

Prysm validator service exit if no pubkeys provided

Related with #36

Validator service exits if not public keys are provided. This is an unexpected behavior that affects the cronjob to fetch the public keys from the web3signer. Consider to push Prysm team to fix this behaviour or start the validator process depending if there are or not public keys

• Track the upstream issue prysmaticlabs/prysm#10294