sigp / blockprint Goto Github PK
View Code? Open in Web Editor NEWBlock fingerprinting for the beacon chain, for client diversity metrics
License: Apache License 2.0
Block fingerprinting for the beacon chain, for client diversity metrics
License: Apache License 2.0
We should support Grandine in the DB. This will be a breaking schema change unfortunately, as it introduces a new column (pr_grandine
). We could provide a simple migration script to upgrade existing DBs so they don't need to be rebuilt from scratch.
At the moment blockprint's DB is not handling sidechains very elegantly.
The database is designed so that a block can be inserted without knowing its parent. This is nice because it provides the ability to stay online processing new blocks, even if some old blocks have been missed due to temporary downtime. There's a background process in background_tasks.py
which is meant to query the API for gaps to fill in, and patch them up.
The problem is that the logic for determining gaps returns some gaps that are impossible for the background task to heal. A gap is currently defined as a slot interval between a block with a parent missing from the DB (end_slot) and the last known block prior to that missing parent (start_slot):
Lines 156 to 163 in c7f570d
Lines 181 to 191 in c7f570d
E.g. the current output from https://api.blockprint.sigp.io/sync/gaps
is:
[
{
"start": 8253079,
"end": 8253086
},
{
"start": 8253031,
"end": 8253045
},
{
"start": 8253127,
"end": 8253151
},
{
"start": 8254502,
"end": 8254511
},
{
"start": 8277096,
"end": 8277097
},
{
"start": 8299646,
"end": 8299647
},
{
"start": 8299650,
"end": 8299651
}
]
Looking at the first gap, we see that the block with missing parent that triggered this must be one at slot 8253087
, which is reorged out (beaconcha.in doesn't even know about it): https://beaconcha.in/slot/8253087. Our Lighthouse nodes saw it though:
Jan 21 18:17:49.412 DEBG Cloned snapshot for late block/skipped slot, block_delay: Some(2.040814714s), parent_root: 0xd429dc371766b1d71fdad731879aafe7c2df990b402fbc5704b29144009cce8f, parent_slot: 8253079, slot: 8253087, service: beacon
Now the interesting thing here is the parent slot, https://beaconcha.in/slot/8253079. It's also empty! In order to heal the gap, we would need to load this parent block at 8253079, which we can't do because it has also been pruned.
In summary, blockprint's gap healing is broken for sidechains of length > 1. I can think of two ways to fix it:
Give the background task the ability to either delete or mark orphaned blocks in the database when: the slot of the missing parent has been finalized as a skipped slot. If we just mark them as orphaned, then we get to keep them in the DB (moar data) but won't block the gap healing process on them. On the other hand, marking them orphaned would require a new database column and a little DB migration (not too bad, given the small number of live blockprint instances).
Currently blockprint's API will return 0 values when querying a section of the database for which history is missing. This can occur if the beacon node backing blockprint has been falling in and out of sync or is re-syncing and lacks a portion of history.
At the moment these gaps can be detected via the /sync/gaps
API, but this is error-prone and not obvious. It would be better if blockprint detected queries affected by the gaps and failed them explicitly so that the user doesn't have to check /sync/gaps
manually. E.g. if there is a gap from slot 3623682 to 3687456 (as there is currently) then blockprint should 404 on queries like /blocks_per_client/113625/113850
rather than returning a response with 0 blocks per client for all clients.
The reason the API is the way it is currently is that the SQL query to detect the gaps takes several seconds to evaluate, because it scans the entire table. If this query can be made to run quickly, then it will be feasible to run it for every query.
I just downloaded the "raw" block rewards for ~6M
main net slots (~1TB of data), which after the classification, stays in ~130GB of "model" (see table below).
The model (size):
2.5G ./model/Prysm
4.3G ./model/Nimbus
31G ./model/Teku
1.2G ./model/Lodestar
92G ./model/Lighthouse
130G ./model
There are a few questions I would like to open here:
Is this an overkilling number of block rewards to train the model?
I'd like to have an accurate enough model, but this seems a bit overkill to me. I'm unsure if you could guide me here based on your experience.
Loading the model currently takes around 50mins, is this an expected duration for the range of time I'm managing?
I checked if there is any concurrency parameter that I could be missing when loading the model (meaning creating an object Classifier
from the given model/folder), but it seems to be sequential by itself.
I also have some doubts about the possible features of the Classifier
. I'm currently using the default ones, and I'll like to know if you guys have better performance using a different combination of features.
Many thanks in advance, and great work with this repo!
Some graffiti regex should only be enabled on e.g. Prater. The ./prepare_training_data.py
script should take a --prater
or --mainnet
flag to switch regex on/off, or a path to a file containing the regex.
It would be cool to have features named and capable of being toggled on and off from the CLI.
It would also be nice to include feature names on the matplotlib graphs, with a 2D cross-section for each pair of features.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.