ethpandaops / dora Goto Github PK
View Code? Open in Web Editor NEWDora the Explorer is a lightweight slot explorer for the ethereum beaconchain
Home Page: https://beaconlight.ephemery.dev/
License: GNU General Public License v3.0
Dora the Explorer is a lightweight slot explorer for the ethereum beaconchain
Home Page: https://beaconlight.ephemery.dev/
License: GNU General Public License v3.0
time="2024-01-06T22:29:59Z" level=error msg="error while deleting validator names from db" error="expected 10000 arguments, got 20000" module=validator_names
Due to the fix of #47, dora usually keeps only 1-2 epochs in memory before the finalization routine cleans them up.
As the most recent epoch is most likely incomplete, it leaves dora with only 1 usable epoch to generate the validator activity aggregation for.
This is not enough to give a reasonable online/offline status for validators.
To fix this, dora should keep validator activity aggregations for finalized epochs in memory for 2-3 more epochs.
This should be enough for a clear online/offline status detection.
The cache does not need to be persisted.
If dora gets restarted, the activity overview will be inaccurate for a few epochs, but that's fine.
I started this kurtosis configuration:
participants:
- el_type: geth
el_image: "192.168.45.152:80/dh/ethpandaops/geth:lightclient-prague-devnet-0"
el_image: "ethpandaops/geth:lightclient-prague-devnet-0"
cl_type: grandine
cl_image: ethpandaops/grandine:feature-electra
count: 3
network_params:
electra_fork_epoch: 1
additional_services: [dora]
snooper_enabled: true
Slots are being proposed but when I try to open the page for a slot I get "Internal server error".
Docker logs indicate:
time="2024-05-13T11:37:56Z" level=error msg="error executing template" error="html/template:slot.html:212:24: no such template \"block_consolidation\"" error type="*template.Error" file=index.go function=Slot info= route=/slot/30
if config.yaml has minimal preset, dora becomes unhappy.
All nodes are still connected, but they all report that they stuck syncing at slot 1.
None of the slots seem to be proposed, even tho they were.
To repro:
kurtosis run github.com/kurtosis-tech/ethereum-package@bbusa/preset --args-file https://github.com/kurtosis-tech/ethereum-package/blob/bbusa/preset/.github/tests/minimal.yaml
The explorer currently can't use clients that do not explicitly follow the specs for some static api calls.
This is due to the initial connection checks implemented in attestantio/go-eth2-client
.
I think these checks should be disabled for the explorer, as the explorer does its own error handling for the api calls it uses.
Especially for early testnets there might be clients that do not follow the specs that closely.
However, they should still be used as much as the explorer can use them.
This issue currently prevents the explorer from using prysm nodes due to this prysm bug:
prysmaticlabs/prysm#12949
Edit: Prysm issue has been resolved
Currently, the explorer fetches blobs directly from the CL node via RPC calls.
This approach won't work once blobs expire and are removed from the CL node.
To allow showing blobs after expiration, it should be possible to store blobs in DB or on an external storage (eg. S3).
This should be optional, so blob storage does not get a dependency to run the explorer.
Noticed a panic in the holesky dora logs:
[negroni] PANIC: runtime error: cannot convert slice with length 0 to array or pointer to array with length 32
goroutine 223955 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:159 +0xc7
panic({0x1326780, 0xc03a70c570})
/usr/local/go/src/runtime/panic.go:884 +0x213
github.com/pk910/dora-the-explorer/handlers.SearchAhead({0x7f14ffc0d7e0, 0xc02118a028}, 0xc00dfb8700)
/src/handlers/search.go:308 +0x210f
net/http.HandlerFunc.ServeHTTP(0xc00dfb8600?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0x446d3c?)
/usr/local/go/src/net/http/server.go:2122 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000000000, {0x7f14ffc0d7e0, 0xc02118a028}, 0xc00dfb8500)
/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7f14ffc0d7e0, 0xc02118a028}, 0x12a5d60?, 0xc01fc0c120)
/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x41e510?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0xc04dc8b900?, 0xc04dc8b978?)
/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1817c80?, 0xc0004e6378?}, 0xc0004e63f0?}, {0x7f14ffc0d7e0, 0xc02118a028}, 0x12a5d60?)
/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0x8?, {0x7f14ffc0d7e0?, 0xc02118a028?}, 0x0?, 0xc04dc8ba50?)
/go/pkg/mod/github.com/urfave/[email protected]/recovery.go:193 +0x86
github.com/urfave/negroni.middleware.ServeHTTP({{0x1815e60?, 0xc0001f6960?}, 0xc0004e63d8?}, {0x7f14ffc0d7e0, 0xc02118a028}, 0x0?)
/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0001f3aa0, {0x1824c40?, 0xc020b340e0}, 0x46e7ce?)
/go/pkg/mod/github.com/urfave/[email protected]/negroni.go:96 +0x125
net/http.serverHandler.ServeHTTP({0xc020cfc660?}, {0x1824c40, 0xc020b340e0}, 0xc00dfb8500)
/usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc038e27dd0, {0x1825568, 0xc0001f3d70})
/usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:3089 +0x5ed
the dora explorer does a good job displaying the consensus slot with the execution payload
https://dora-holesky.pk910.de/slot/2
however, it does not do it for genesis. the structure should be the same though?
Whisk SSLE (Single Secret Leader Election) is a new Ethereum protocol feature being tested now. (ref: https://ethresear.ch/t/whisk-a-practical-shuffle-based-ssle-protocol-for-ethereum/11763)
The point of the feature is to identity of the next proposer private until its block is published. This means:
@parithosh has deployed a testnet for whisk https://github.com/ethpandaops/whisk-testnets/ with this explorer. BTW great work on the explorer, love it so far! ❤️
However the explorer stalls indexing at the Whisk fork epoch since it cannot poll duties. More precisely, the beacon node returns an error to calls to eth/v1/validator/duties/proposer/
since it cannot serve that data.
INFO[0008] synchronizing epoch 0 module=synchronizer
WARN[0008] could not lazy load proposer duties for epoch 1501: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1501, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]} client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1501 client=default module=indexer
WARN[0008] could not lazy load proposer duties for epoch 1503: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1503, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]} client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1503 client=default module=indexer
WARN[0008] could not lazy load proposer duties for epoch 1502: error retrieving proposer duties: url: http://localhost:5052/eth/v1/validator/duties/proposer/1502, error-response: {"code":500,"message":"UNHANDLED_ERROR: BeaconStateError(NoProposerShuffleAfterWhisk)","stacktraces":[]} client=default module=indexer
WARN[0008] could not find proposer duties for epoch 1502 client=default module=indexer
INFO[0010] received block 1503:48117 [0x6a322ff15c787ce63c92a0c062f4ce462263cb8d58737e7829bc25d40ca88473] stream client=default module=indexer
INFO[0015] index page called
It would be great to have this explorer be compatible with Whisk for ease of debugging on devnets and to start iterating on how explorers would deal with post-whisk world with proposer privacy.
Please let me know what ideas you have to deal with case, I will suggest some ideas below
Add a config value whisk_epoch
after which the indexer fills the proposer duties response with dummy values (i.e. the 0 validator). This won't require any changes to the UI, API, etc and should allow to continue indexing
Add a config value whisk_epoch
after which indexer changes the render of proposer duties.
Feature request:
The explorer should be able to show eth transaction details on the block details page.
Current status:
The explorer only displays the transaction count, but no further transaction details.
Technically the signed transactions are available for post-bellatrix blocks without adding further complexity by calling a EL RPC.
However, they're not yet decoded or visualized.
I think it'd be great to get some basic details (hash, from/to addr, amount, & call data) displayed for each transaction in the block details.
Further transaction details like receipt information, events, internal transactions or tx traces should not be included for the first implementation.
Showing that stuff would need some EL RPC interaction, which introduces additional complexity that might not be worth adding.
Decoding the transactions should be easy: example
I think that's a good first issue for anyone who wants to join development for this explorer :)
Anyone who contributed to beaconcha.in before will probably feel familiar with the structure of this lightweight explorer ;)
Feature Request:
Add a list of attester / proposer slashings / track slashings in db
Dora currently stores blob sidecars with commitment
as primary key.
This causes issues when commitments are re-used in multiple blob transactions.
In this situation, dora would currently override the blob entry from the older block.
It's effectively removing the blob sidecar link from the old block, which is obviously wrong.
The attestantio/go-eth2-client
implementation currently does not return the metadata fields supplied via various endpoints.
But the explorer needs the dependent_root
field from /eth/v1/validator/duties/proposer/{epoch} call.
So there is currently a workaround code that does the getProposerDuties on its own.
This workaround code should be removed once the upstream library has support for these metadata fields.
Feature Request:
Dora should index the EL block extra data field and allow pattern search via the filtered slots page
This is not implemented yet and will cause some index inconsistency when it happens.
However, I guess such big reorgs normally shouldn't happen ever.
I'm trying to start the explorer for a Devnet based on the Gnosis Chain Preset. Currently, it fails with the following message:
time="2023-10-03T18:27:49Z" level=fatal msg="error reading config file: tried to use unknown chain-preset: gnosis"
Possible Solution:
conformant MEV relays host these data APIs:
https://flashbots.github.io/relay-specs/#/
it would be nice if there was an integration in dora.
thinking something like:
this is need bigtable sql sir ?
Dora should be able to track & display deposits.
The explorer should be able to load the current network specs from the connected beacon nodes.
This will make the explorer easier to run on custom devnets.
time="2024-02-02T12:45:44Z" level=info msg="http server listening on 0.0.0.0:8080"
time="2024-02-02T12:45:44Z" level=info msg="received block 0:0 [0x71ca733aff61e91d304fe23b869010fd267a4a3931a736c6729a7657ca670507] warmup, head" client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:45:44Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:45:54Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer
time="2024-02-02T12:46:04Z" level=warning msg="indexer client error: could not find parent header 0x0000000000000000000000000000000000000000000000000000000000000000, retrying in 10 sec..." client=cl-1-lighthouse-geth module=indexer
the client is stuck with the genesis block:
~# curl http://127.0.0.1:32832/eth/v1/beacon/headers
{"execution_optimistic":false,"finalized":false,"data":[{"root":"0x71ca733aff61e91d304fe23b869010fd267a4a3931a736c6729a7657ca670507","canonical":true,"header":{"message":{"slot":"0","proposer_index":"0","parent_root":"0x0000000000000000000000000000000000000000000000000000000000000000","state_root":"0xfe842cf4d68b07159db26cfca3ebfe14b030d48438a6e5e9745eea78f6b8153f","body_root":"0x74b4bb048d39c75f175fbb2311062eb9867d79b712907f39544fcaf2d7e1b433"},"signature":"0x000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"}}]}
The explorer currently consumes quite much memory for operation.
Biggest memory consumptions:
Especially the first point should be improved.
The explorer currently does up to 2 validator set requests in parallel, so it's consuming up to 3GB just for parsing some giant json
The explorer currently grabs the finalization status too early during epoch transition and therefore shows finalized epochs an epoch later than it should.
On kaustinen at least, it seems that the slot number is actually the block number (using a slot number of e.g. 74 yields the execution data for block number 74).
https://dora.kaustinen-testnet.ethpandaops.io/slot/74
But the block number itself is reported to be 116, which is wrong. If you look at the slots before and after, the block values are jumping widely, which has the result of the block numbers being AHEAD of the slot numbers. This should not happen on a testnet who merged at genesis.
The explorer sees the forked off chain from nimbus clients as canonical chain here.
This is a non-persistent issue, which already resolved itself a block after.
The explorer selects the canonical chain head on the fly based on the heads from all connected clients.
It looks like the nimbus head was selected here because it has the highest slot number. This logic should be improved.
Dear @ethpandaops
We used Dora for our project: Canxium, it's really awesome, could you please give us your wallet, we'd like to send you some CAU coin as a donation (metamask wallet is fine)
Best Regards.
Neo Krypt.
I've noticed some issues with the orphaned block detection:
https://beaconlight-devnet.pk910.de/epoch/7346
https://beaconlight-devnet.pk910.de/epoch/7737
Looks like the reorg detection marks too many blocks as orphaned..
Needs further debugging
Dora keeps getting stuck in rare situations.
This seems to be related to bad client connections, which makes dora running into a deadlock.
The dora core logic (including client pool & block cache) should be reworked to make such deadlocks impossible
(ie. use channels and contexts instead of locking)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.