status-im / nimbus-eth2 Goto Github PK
View Code? Open in Web Editor NEWNim implementation of the Ethereum Beacon Chain
Home Page: https://nimbus.guide
License: Other
Nim implementation of the Ethereum Beacon Chain
Home Page: https://nimbus.guide
License: Other
During the call today, we discussed repair and compared it with sync. I mentioned that one way of approaching the problem is to split out unverified blocks and attestations into a separate space - the idea is similar to tainted strings or a firewalled network - basically, we would uphold stronger invariants in a "safe space" and make sure to guard the gate between the two..
Necessary criteria for entering the safe space would be at least the following (Item here is either an attestation or block):
Another way to express those criteria more succinctly is that we should only permit items in there that have slashing conditions attached to them, in case they turn out to be malicious.
Pro:
Con:
Explored in this branch https://github.com/status-im/nim-beacon-chain/tree/fix-sim-init-windows
The shards and committee were never populated.
Latest changes removed them, which might fix the issue.
Unfortunately this is pending a fix or workaround for #78
in state sim, with --attesterRatio=1 --slots=4096
everyone should be attesting - however, the first validator is getting penalized - investigate this:
General idea is to switch beacon_node
backend to libp2p
using daemon and develop a simple protocol on top using SSZ as serialization.
Further, the idea is to introduce a management layer that contains logic for dealing with retrying requests and coordinating peer scoring. Basically, the attestation and block pools signal the hashes they need and a separate layer decides on the logic to fetch these from the peer layer (how many concurrent requests, when to retry the same block). When blocks arrive, either from broadcasts or requests, they should flow into the pool the same way.
libp2p
via daemon (postponed)When joining the testnet, client will be behind. We will regularly restart the testnet in the beginning, thus we primarily need to have the capability to catch up by "full sync" - downloading all blocks. The other case where blocks are needed is when an attestation or block is received, and the dependent blocks are not (lost in translation, missing history, unknown fork etc)
After validating or proposing blocks, these will be (naively) gossiped all other participants so they can count votes and decide on forks. The most simple implementation idea seems to be to publish attestations with a single signature, then aggregate lazily as needed (for example when proposing a block).
When switching to libp2p
, make sure these issues are covered and used correctly, as a minimum:
Forks start with the latest finalized block and build a tree of possible futures from there. The idea is to manage known blocks and attestations as a collection, and take action to fill out that collection as needed.
There are many race conditions that all need to be handled gracefully, and the code should have room to modify the strategy for handling these:
One problem to consider is that of worst case performance, in case of malicious blocks being posted by validators (for example, lots of unviable forks / blocks causing data structure and network traffic growth)
Adding and removing validators is somewhat in flux, so for now the plan is to not use the ETH1 contract for this feature. Initially, we'll just publish JSON files with validator data (priv key etc) and manage overlap socially. The majority will likely be used in pre-configured beacon nodes running on a server, while some will be reserved for developers to play with.
Potential issues include two people running the same validator - this is a feature as it will help us discover issues when this happens (for example if we receive an attestation signed by our own key that we did not send, this is a warning sign that the private key is being reused).
We'll initially deploy one or more boot nodes on a server, each hosting a number of validators. The general idea is to restart the testnet frequently. People wanting to connect will get genesis and validator information from the server via.. whatever (http listing).
Don't wanna run a big network just yet ๐
The general idea is to follow spec releases by updating every time there's a new upstream release.
nim-beacon-chain
according to spec version it supportsFrom time to time the simulation skips slots.
Example in https://gist.github.com/mratsim/1060f973b38c915db3172c27cc7e8e5f
Slot 6 and 8 were skipped. This can also happen later, in the same run slot 25 and 27 were skipped.
On Linux x86-64
Similar to #89 #97 we have an issue during epoch transition, this times after 64 slots, for epoch 144115188075855873
Full logs available in https://gist.github.com/mratsim/a6539c049dfb49a638327b9db5cec44a.
There are 2 logs, one with multiple nodes desynced, one with only a single node desynced.
The following log is after merging #103 for epochs printing.
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=338 slot=121 validator=96d7e599 node=0
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=296 slot=114 validator=8e724e86 node=5
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=326 slot=119 validator=0a89ffbe node=2
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=350 slot=123 validator=91792a76 node=7
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [AssertionError]
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=344 slot=122 validator=842ef0e2 node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=356 slot=124 validator=861b6b0b node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=362 slot=125 validator=928a8a8e node=8
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=374 slot=127 validator=9090c4be node=1
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
INF 2019-02-11 16:09:08+01:00 Scheduling block proposal thread=0 fromNow=380 slot=128 validator=968a8996 node=8
beacon_node.nim(435) beacon_node
beacon_node.nim(399) processBlocks
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(328) :anonymous
beacon_node.nim(306) scheduleEpochActions
validator.nim(95) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(95, 10) `epoch <= next_epoch` Previous epoch: 144115188075855872, epoch: 144115188075855874, Next epoch: 144115188075855873 [Assert
Terminated
As evidenced by 43554ad in PR #80, many places are still using domain 0 in hash_to_G2 / sig verification.
This needs to be fixed. It is also very probably that several parts of Eth2 specs does not use domain.
cc @cheatfate
Currently we are using an IntSet for the attester_bitfield field. However this is not ideal.
type MyBitfield = object
data: set[int16]
len: int16
We can be sure that it works for our purposes but extra implementation, testing and maintenance time.
Python implementation is a short 36 lines here. Tests.
Note that the beacon chain specs mentions len(attester_bitfield) == ceil_div8(len(attestation_indices))
with ceil_div8 = (x + 7) // 8
, This checks that the byte length of the bitfield is equal to the number of attesters divided by 8 and rounded up to the nearest multiple of 8. An alternative and arguably clearer implementation can be derived from Nimbus ceil32
With regards to ethereum/eth2.0-specs#593
When messages are received on the network, we should not propagate them unless they're "good" (to be defined).
libp2p allows blacklisting peers by peer id (which is secured by the above feature, https://github.com/libp2p/go-libp2p-pubsub/blob/master/blacklist_test.go#L38-L64).
There also exists a validation api (https://github.com/libp2p/go-libp2p-pubsub/blob/bfd65a2f6b810c5b4ad2cfe6bb9cc792fd7a0171/floodsub_test.go#L360) to filter messages by content - it is making its way into the daemon api as well: libp2p/go-libp2p-daemon#71
How do we make use of this information? should it affect peer scoring?
INF 2019-03-08 01:58:04-06:00 Storing state tid=11707 stateRoot=34568B2765DF7853F87D9DEF88F27A83DDD421FE8C2F44D6EA78044DC704871C stateSlot=8192 node=9
INF 2019-03-08 01:58:04-06:00 Updated head tid=11707 headBlockRoot=44E30AF2 stateRoot=34568B27 stateSlot=8192 node=9
DBG 2019-03-08 01:58:04-06:00 Advancing state past slot gap tid=11707 stateSlot=8192 targetSlot=8193 node=9
INF 2019-03-08 01:58:04-06:00 Scheduling next slot start action block tid=11707 fromNow=5 slot=8194 node=9
DBG 2019-03-08 01:58:04-06:00 No new head found tid=11707 blockRoot=44E30AF2 stateRoot=34568B27 stateSlot=8192 node=9
DBG 2019-03-08 01:58:04-06:00 Advancing state past slot gap tid=11707 stateSlot=8192 targetSlot=8193 node=9
beacon_node.nim(667) beacon_node
beacon_node.nim(621) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(417) :anonymous
asyncmacro2.nim(306) makeAttestation
asyncmacro2.nim(36) makeAttestation_continue
beacon_node.nim(270) makeAttestationIter
beaconstate.nim(278) get_block_root
system.nim(3790) failedAssertImpl
system.nim(3783) raiseAssert
system.nim(2830) sysFatal
[[reraised from:
beacon_node.nim(667) beacon_node
beacon_node.nim(621) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncfutures2.nim(370) cb
]]
Error: unhandled exception: /home/arnetheduck/status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim(278, 12) `state.slot <= slot + 8192`
Async traceback:
beacon_node.nim(667) beacon_node
beacon_node.nim(621) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
beacon_node.nim(417) :anonymous
asyncmacro2.nim(306) makeAttestation
asyncmacro2.nim(36) makeAttestation_continue
beacon_node.nim(270) makeAttestationIter
beaconstate.nim(278) get_block_root
system.nim(3790) failedAssertImpl
system.nim(3783) raiseAssert
system.nim(2830) sysFatal
I suspect it's related to us not finalizing stuff, and at the same time reaching the limit of some state history field
Node #1 disconnect from the master node due to a TcpError but still crashes after instead of handling it gracefully.
Node 0 was communicating with another node (not sure about the port 37580) rdisconnected the peer but still crashed
same data as #150 - latest_crosslinks are being updated only for the first few shards - investigate why this is. notably, there's only 64 validators in that run which might be related
Sig aggregation has OK times but public key aggregation is very slow: status-im/nim-blscurve#13.
Pubkey aggregation might not be needed though. To be checked.
print what folder is being used (~/.cache/nimbus/...
) exactly, so user knows where to look
it would probably be good if benchmarks were expressed in terms of spec/crypto.nim
..
NOT 2019-03-26 14:50:04+01:00 Invalid block tid=3075 blck="(slot: 7, previous_block_root: \"52C0B4F5\", state_root: \"06052C4E\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits
_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6d48dda\")" blockRoot=9547363C node=1
Traceback (most recent call last)
beacon_node.nim(771) beacon_node
beacon_node.nim(691) start
beacon_node.nim(664) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
asyncmacro2.nim(36) proposeBlock_continue
chronicles.nim(167) proposeBlockIter
digest.nim(35) shortLog
hash.nim $
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
as of https://github.com/status-im/nim-beacon-chain/commits/master
Changelog: https://notes.ethereum.org/TQCF7Q07RI2tlP8Vmiv3RQ?view
validators
and dynasty
to get_active_validator_indices
dynasty
to get_new_shuffling
ActiveState.recent_block_hashes
dictating orderActiveState.recent_block_hashes
parent_hashes
calculationcurrent_checkpoint
get_new_shuffling
takes seed
as an arg and passes it to shuffle
Attestation
validity condition -- slot > block.slot_number - EPOCH_LENGTH
ShardAndIndices
to ShardAndCommittee
ShardAndCommittee.validators
to ShardAndCommittee.committee
INF 2019-03-27 02:40:06+01:00 New testnet genesis data received. Starting with a fresh database. tid=12761
Failed to download key
cannot open: /home/swader/.cache/nimbus/BeaconNode/testnet0/network.jsonswader@swader-N76VZ:~/repos/nimbus/vendor/nim-beacon-chain$
This when restarting testnet join without cache etc. rm -rf ~/.cache/nimbus et al.
DBG 2019-03-26 09:34:21+01:00 Block already exists tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 16, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=84786b22 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 16, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=8af56524 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=ac1f99bc node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 17, beacon_block_root: \"445C5B9B\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 1, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
INF 2019-03-26 09:34:21+01:00 Scheduling next slot start action block tid=4775 fromNow=18446744073709533 slot=34 node=6
INF 2019-03-26 09:34:21+01:00 Attestation sent tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b68d2663 validator=86d386aa node=6
INF 2019-03-26 09:34:21+01:00 Scheduling next slot start action block tid=4775 fromNow=18446744073709539 slot=35 node=6
INF 2019-03-26 09:34:21+01:00 GossipSub Peer connected tid=4794 peer=Node[127.0.0.1:50006] node=4
INF 2019-03-26 09:34:21+01:00 Block proposed tid=4775 blck="(slot: 18, previous_block_root: \"723CF95D\", state_root: \"4F2093C3\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 2, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b13d1c9a\")" blockRoot=4300C585 idx=62 validator=86d386aa node=6
DBG 2019-03-26 09:34:21+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4792 remote=Node[127.0.0.1:50006] node=7
DBG 2019-03-26 09:34:21+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4794 remote=Node[127.0.0.1:50006] node=4
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 20, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b966bba9 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 20, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4794 attestationData="(slot: 16, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=92484f3c node=4
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=85894d4a node=6
DBG 2019-03-26 09:34:21+01:00 Growing attestation pool tid=4775 attestationSlot=22 startingSlot=0 node=6
ERR 2019-03-26 09:34:21+01:00 dispatchMessages failed topics="rlpx" tid=4794 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(36) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncmacro2.nim(306) emit\n asyncmacro2.nim(36) emit_continue\n gossipsub_protocol.nim(61) emitIter\n serialization.nim(65) :anonymous\n beacon_node.nim(666) :anonymous\n beacon_node.nim(638) onAttestation\n attestation_pool.nim(173) add\n attestation_pool.nim(71) validate\n validator.nim(150) get_crosslink_committees_at_slot\n system.nim(3879) failedAssertImpl\n system.nim(3872) raiseAssert\n system.nim(2918) sysFatal\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(39) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncfutures2.nim(324) read\n ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch:INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
2, Next epoch: 1\nException type:" node=4
INF 2019-03-26 09:34:21+01:00 GossipSub Peer disconnected tid=4794 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=4
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=9796577f node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=1 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=a3066e4b node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=2 node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4775 attestationData="(slot: 19, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=b950bc65 node=6
INF 2019-03-26 09:34:21+01:00 Attestation resolved tid=4775 attestationData="(slot: 19, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" validations=3 node=6
INF 2019-03-26 09:34:21+01:00 Attestation sent tid=4775 attestationData="(slot: 22, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=99567d7e validator=91887afb node=6
DBG 2019-03-26 09:34:21+01:00 Attestation received tid=4794 attestationData="(slot: 18, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 2, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=9796577f node=4
DBG 2019-03-26 09:34:21+01:00 Block received tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
ERR 2019-03-26 09:34:21+01:00 dispatchMessages failed topics="rlpx" tid=4794 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(36) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncmacro2.nim(306) emit\n asyncmacro2.nim(36) emit_continue\n gossipsub_protocol.nim(61) emitIter\n serialization.nim(65) :anonymous\n beacon_node.nim(666) :anonymous\n beacon_node.nim(638) onAttestation\n attestation_pool.nim(173) add\n attestation_pool.nim(71) validate\n validator.nim(150) get_crosslink_committees_at_slot\n system.nim(3879) failedAssertImpl\n system.nim(3872) raiseAssert\n system.nim(2918) sysFatal\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(39) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncfutures2.nim(324) read\n ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=4
INF 2019-03-26 09:34:21+01:00 GossipSub Peer disconnected tid=4794 peer=Node[127.0.0.1:50006] reason=ClientQuitting node=4
INF 2019-03-26 09:34:23+01:00 Preparing for fork choice tid=4775 connectedPeers=3 node=6
DBG 2019-03-26 09:34:23+01:00 Replaying state transitions tid=4775 ancestors=1 prevStateSlot=0 stateRoot=246EF52C stateSlot=12 node=6
DBG 2019-03-26 09:34:24+01:00 Advancing state with empty slots tid=4794 stateSlot=6 targetSlot=10 node=4
INF 2019-03-26 09:34:24+01:00 Storing state tid=4775 stateRoot=246EF52C stateSlot=0 node=6
DBG 2019-03-26 09:34:24+01:00 No head update this time tid=4775 headBlockRoot=723CF95D headBlockSlot=13 node=6
NOT 2019-03-26 09:34:24+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=4775 expectedEpoch=4 expectedSlot=37 fromNow=3 headSlot=13 node=6
INF 2019-03-26 09:34:24+01:00 Scheduling next slot start action block tid=4775 fromNow=3 slot=38 node=6
DBG 2019-03-26 09:34:24+01:00 Attestation received tid=4775 attestationData="(slot: 23, beacon_block_root: \"723CF95D\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 3, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=8033bc50 node=6
ERR 2019-03-26 09:34:24+01:00 dispatchMessages failed topics="rlpx" tid=4775 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(39) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(36) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncmacro2.nim(306) emit\n asyncmacro2.nim(36) emit_continue\n gossipsub_protocol.nim(61) emitIter\n serialization.nim(65) :anonymous\n beacon_node.nim(666) :anonymous\n beacon_node.nim(638) onAttestation\n attestation_pool.nim(173) add\n attestation_pool.nim(71) validate\n validator.nim(150) get_crosslink_committees_at_slot\n system.nim(3879) failedAssertImpl\n system.nim(3872) raiseAssert\n system.nim(2918) sysFatal\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(39) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(39) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncfutures2.nim(324) read\n ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=6
INF 2019-03-26 09:34:24+01:00 GossipSub Peer disconnected tid=4775 peer=Node[127.0.0.1:50008] reason=ClientQuitting node=6
DBG 2019-03-26 09:34:28+01:00 Block already exists tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block received tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block already exists tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block received tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Block already exists tid=4775 blck="(slot: 6, previous_block_root: \"1AF2EF4D\", state_root: \"091C0886\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 1, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b4dc4f9d\")" blockRoot=9817B58C node=6
DBG 2019-03-26 09:34:28+01:00 Attestation received tid=4775 attestationData="(slot: 16, beacon_block_root: \"DB12D744\", source_epoch: 18446744073709551615, target_root: \"00000000\", source_root: \"00000000\", shard: 0, previous_crosslink_epoch: 1, previous_crosslink_data_root: \"00000000\", crosslink_data_root: \"00000000\")" signature=abc75cff node=6
ERR 2019-03-26 09:34:28+01:00 dispatchMessages failed topics="rlpx" tid=4775 err="/home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(39) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(36) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncmacro2.nim(306) emit\n asyncmacro2.nim(36) emit_continue\n gossipsub_protocol.nim(61) emitIter\n serialization.nim(65) :anonymous\n beacon_node.nim(666) :anonymous\n beacon_node.nim(638) onAttestation\n attestation_pool.nim(173) add\n attestation_pool.nim(71) validate\n validator.nim(150) get_crosslink_committees_at_slot\n system.nim(3879) failedAssertImpl\n system.nim(3872) raiseAssert\n system.nim(2918) sysFatal\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(39) dispatchMessages_continue\n rlpx.nim(518) dispatchMessagesIter\n rlpx.nim(265) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(39) emit_thunk_continue\n rlpx.nim(769) emit_thunkIter\n asyncfutures2.nim(324) read\n ]#\nException message: /home/beta/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(150, 12) `epoch <= next_epoch` Previous epoch: 18446744073709551615, epoch: 2, Next epoch: 1\nException type:" node=6
INF 2019-03-26 09:34:28+01:00 GossipSub Peer disconnected tid=4775 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=6
The cutoffs function description is the followinf:
An algorithm that we use to split up validators into groups at the start of every epoch, determining at what height they can make attestations and what shard they are making crosslinks for.
The schema:
A full Python implementation:
SHARD_COUNT = 1024
ETH_SUPPLY_CAP = 2**27
DEPOSIT_SIZE = 32
MAX_VALIDATOR_COUNT = ETH_SUPPLY_CAP // DEPOSIT_SIZE
EPOCH_LENGTH = 64
SLOT_DURATION = 8
MIN_COMMITTEE_SIZE = 128
END_EPOCH_GRACE_PERIOD = 8
def get_cutoffs(validator_count):
height_cutoffs = [0]
# EPOCH_LENGTH // phi
cofactor = 39
STANDARD_COMMITTEE_SIZE = MAX_VALIDATOR_COUNT // SHARD_COUNT
# If there are not enough validators to fill a minimally
# sized committee at every height, skip some heights
if validator_count < EPOCH_LENGTH * MIN_COMMITTEE_SIZE:
height_count = validator_count // MIN_COMMITTEE_SIZE or 1
heights = [(i * cofactor) % EPOCH_LENGTH
for i in range(height_count)]
# If there are enough validators, fill all the heights
else:
height_count = EPOCH_LENGTH
heights = list(range(EPOCH_LENGTH))
filled = 0
for i in range(EPOCH_LENGTH - 1):
if not i in heights:
height_cutoffs.append(height_cutoffs[-1])
else:
filled += 1
height_cutoffs.append(filled * validator_count // height_count)
height_cutoffs.append(validator_count)
# For the validators assigned to each height, split them up
# into committees for different shards. Do not assign the
# last END_EPOCH_GRACE_PERIOD heights in an epoch to any shards.
shard_cutoffs = [0]
for i in range(EPOCH_LENGTH - END_EPOCH_GRACE_PERIOD):
size = height_cutoffs[i+1] - height_cutoffs[i]
shards = (size + STANDARD_COMMITTEE_SIZE - 1) // STANDARD_COMMITTEE_SIZE
pre = shard_cutoffs[-1]
for j in range(1, shards+1):
shard_cutoffs.append(pre + size * j // shards)
return height_cutoffs, shard_cutoffs
(h, s) = get_cutoffs(16)
# Note: Python 2 because Apple/Mac ...
print height # [0, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16]
print len(height) # 65
print shard # [0, 16]
The Python implementation does not split like in the example
The example may be a simplification that shows the spirit of the spec but no sharding occurs because 16 validators is too small.
Nim implementation:
https://github.com/status-im/nim-beacon-chain/blob/585072ae154e3d10276b51397e707628a401c096/beacon_chain/private/helpers.nim#L35-L79
# to run from ./build
import ../beacon_chain/private/helpers
import ../beacon_chain/datatypes
let a = 16
var seed: Blake2_256_Digest
echo getShuffling(seed, a) # @[14, 9, 13, 10, 3, 6, 12, 1, 2, 0, 5, 15, 8, 7, 11, 4]
let (height, shard) = getCutoffs(a)
echo height # @[0, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16]
echo height.len # 65
echo shard # @[0, 16]
[arnetheduck@tempus research]$ nim c --opt:speed state_sim ; ./state_sim --attesterRatio=1 --slots=128 --validators=1000
...
confutils.nim(362) state_sim
state_sim.nim(74) CLI
testutil.nim(82) addBlock
validator.nim(273) get_beacon_proposer_index
validator.nim(243) get_crosslink_committees_at_slot
validator.nim(132) get_shuffling
system.nim(2830) sysFatal
Error: unhandled exception: index out of bounds [IndexError]
This was a bit tricky are many ranges are implicitly assumed in the code at the moment.
To allow simulation on more than 99 validators and 9 validators node you need to change 5 things:
tests/simulation/start.sh
validator_keygen.nim
Example commits in https://github.com/status-im/nim-beacon-chain/tree/allow-more-than-99-validators
Hit this after following these directions and running for a day or so.
~/src/status-im/nimbus/vendor/nim-beacon-chain $ git rev-parse HEAD
c9e0fed4eac43e1867465a88cf9d918b6dd8c8fe
~/src/status-im/nimbus $ git rev-parse HEAD
655a997559664b53424673b96f3818201f51c96d
$ brew info rocksdb
rocksdb: stable 5.18.3 (bottled)
Embeddable, persistent key-value store for fast storage
https://rocksdb.org/
/usr/local/Cellar/rocksdb/5.18.3 (107 files, 59.4MB) *
Poured from bottle on 2019-03-29 at 21:08:32
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/rocksdb.rb
==> Dependencies
Required: gflags โ, lz4 โ, snappy โ
==> Analytics
install: 1,278 (30 days), 3,499 (90 days), 21,878 (365 days)
install_on_request: 541 (30 days), 1,532 (90 days), 8,767 (365 days)
build_error: 0 (30 days)
len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD [2/1824]
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_
len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Unresolved block tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5066, previous_block_root: \"A96F8DBD\", state_root: \"00094835\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"aa4b196c\")" blockRoot=3CD9C0C4
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5070, previous_block_root: \"167A5A61\", state_root: \"96DC41A7\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b444eb3a\")" blockRoot=41EA2013
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Unresolved block tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5056, previous_block_root: \"0FE1D1C3\", state_root: \"2B530C50\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"951cd66e\")" blockRoot=6DE1F47F
DBG 2019-03-31 06:50:03-07:00 Unresolved block tid=5348663 blck="(slot: 5056, previous_block_root: \"0FE1D1C3\", state_root: \"2B530C50\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"951cd66e\")" blockRoot=6DE1F47F
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5055, previous_block_root: \"35B06FD7\", state_root: \"F06CA02F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6b82eef\")" blockRoot=0FE1D1C3
DBG 2019-03-31 06:50:03-07:00 Unresolved block tid=5348663 blck="(slot: 5055, previous_block_root: \"35B06FD7\", state_root: \"F06CA02F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b6b82eef\")" blockRoot=0FE1D1C3
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5074, previous_block_root: \"F2D2ABEC\", state_root: \"C0B2D833\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"855cbf22\")" blockRoot=BA067036
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5072, previous_block_root: \"9B5891B0\", state_root: \"CC15B79D\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b3bb91ef\")" blockRoot=F2D2ABEC
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5071, previous_block_root: \"41EA2013\", state_root: \"47651ED4\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b83ddb59\")" blockRoot=9B5891B0
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5070, previous_block_root: \"167A5A61\", state_root: \"96DC41A7\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b444eb3a\")" blockRoot=41EA2013
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5069, previous_block_root: \"3CD9C0C4\", state_root: \"F65176A0\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"81f10dc4\")" blockRoot=167A5A61
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5066, previous_block_root: \"A96F8DBD\", state_root: \"00094835\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"aa4b196c\")" blockRoot=3CD9C0C4
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5065, previous_block_root: \"70C2BCBB\", state_root: \"5912FAEA\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5065, previous_block_root: \"70C2BCBB\", state_root: \"5912FAEA\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad2d366d\")" blockRoot=A96F8DBD
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5062, previous_block_root: \"D1410D88\", state_root: \"14FD87E9\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"b05b89d4\")" blockRoot=70C2BCBB
DBG 2019-03-31 06:50:03-07:00 Block received tid=5348663 blck="(slot: 5058, previous_block_root: \"6DE1F47F\", state_root: \"0A4D1236\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"929b9217\")" blockRoot=D1410D88
DBG 2019-03-31 06:50:03-07:00 Slot start tid=5348663 lastSlot=5525 scheduledSlot=5526 slot=5526
DBG 2019-03-31 06:50:03-07:00 Advancing state with empty slots tid=5348663 stateSlot=5525 targetSlot=5526
Error: unhandled exception: /Users/nathanielgentile/src/status-im/nimbus/vendor/nim-beacon-chain/beacon_chain/spec/validator.nim(152, 12) `previous_epoch <= epoch` Previous epoch: 689, epoch: 688 (slot: 5507), Next epoch: 691 [AssertionError]
I don't have the entire log due to tmux history limit, but here is a more complete log: full-log
With regards to ethereum/consensus-specs#593
Gossipsub messages are signed by the peer id / peer public key (libp2p/specs#140), meaning it's possible to link an attestation to a peer id, and therefore trivially lookup transport details of a validator.
The peer ID is a hash of the public key. If youโre using a recoverable signature, you should be able to match on pubkey, but once you have the pubkey, the conversion to a peer ID is trivial
cc @zah
Change in BLS scheme #80 (?) broke the simulation in tests/simulation/start.sh
.
Reproduced on Mac and Linux.
What is strange is that BLS passes all the tests including eth2.0-tests with the EF test generator and it also passes the classic beacon-chain tests.
beacon_node.nim(416) beacon_node
trusted_state_snapshots.nim(33) createStateSnapshot
beaconstate.nim(214) get_initial_beacon_state
beaconstate.nim(87) process_deposit
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /Users/tesuji/Programming/Status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim(87, 12) `state.validator_registry[index].withdrawal_credentials ==
withdrawal_credentials` [AssertionError]
Terminated: 15
An additional issue on top of #79 that impacts everyone and not only windows:
DBG 2019-02-07 12:07:34+01:00 Attaching validator thread=0 idx=0 pubKey=02c6d059784c69da51cc5f6c878a0852722186fce95f6644e86631e7f8ab332b7a05c1f8168d23a8399b107d4f33db2e validator=02c6d059 node=0
DBG 2019-02-07 12:07:34+01:00 Attaching validator thread=0 idx=1 pubKey=0b54490fe9f66437ce21e2f6bfb4324e4970b347d5543de47250c72fcf7f2f74e077efd5d45ed3b94b31a5c9d3953d8f validator=0b54490f node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=2 pubKey=0ee02e4857640286c946dac512e09b9e2f045431262f942f90aa6d7a12f8ddb4629b74153d8cc64fb965bbb5231888b9 validator=0ee02e48 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=3 pubKey=13921516d60bd2fd107e888afcfb363191d9d1ab1594f8ef57db8d016c7884f5252b8df2be01f3c11d3f4e7b89a4e37d validator=13921516 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=4 pubKey=8a538ee663cb9691e7f0dc82d88e3c0dbf4ef92c77581e93f98a81879a865a2db006523ebea52752fe06b80b145a63b1 validator=8a538ee6 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=5 pubKey=8473b922c54af034d3875ace14b3abc0c2ff994e1b05ef2ce84da908ae2c4c5f1a6d3cf0a62ba22a483361c9035e8bb9 validator=8473b922 node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=6 pubKey=13830d6dd70c9daa9dcfeecc3bf920a914462c0ad0e34c176bd14b30567e2c7738cd25306f9611a3b5bdee69e8adafa0 validator=13830d6d node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=7 pubKey=85ea693dc32596b6763f6b7f6cbcde0d5b34c84cc8d4e54c3bd2cf2520a42ff4c37086bcfd32077ca12bffc1bb8d5c6f validator=85ea693d node=0
DBG 2019-02-07 12:07:35+01:00 Attaching validator thread=0 idx=8 pubKey=88e100a98e4f3b28f8128a02debab5b7e4cc7716642d797f8f36679fe5be170af9965e4201fc1a735bfe2af4ccc7c793 validator=88e100a9 node=0
INF 2019-02-07 12:07:35+01:00 Local validators attached thread=0 count=9 node=0
DBG 2019-02-07 12:07:35+01:00 Scheduling epoch actions thread=0 epoch=2 node=0
Current epoch: 2
Next epoch: 2
Current epoch: 144115188075855872 # 2^57 ~ย Genesis block starts at 2^63 / 64 = 2^57
Next epoch: 1
beacon_node.nim(431) beacon_node
beacon_node.nim(393) processBlocks
beacon_node.nim(302) scheduleEpochActions
validator.nim(89) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /Users/tesuji/Programming/Status/nim-beacon-chain/beacon_chain/spec/validator.nim(89, 10) `previous_epoch <= epoch` [AssertionError]
This was introduced following the latest spec changes, probably the introduction of get_crosslink_committees_at_slot
and the removal of get_shard_committees_at_slot
.
As of https://github.com/status-im/nim-beacon-chain/tree/e4c10a31c9ab147ff0cffc3c8b2d120ddae69806
Errors:
INF 2019-03-26 09:34:28+01:00 Block resolved tid=4794 blck="(slot: 15, previous_block_root: \"1270EF65\", state_root: \"D9DEA76F\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, transfers_len: 0, signature: \"ad15fd9f\")" blockRoot=EC2C20F8 node=4
INF 2019-03-26 09:34:28+01:00 Forward sync imported blocks tid=4794 badBlocks=1 bodies=14 goodBlocks=13 headers=14 roots=14 node=4
INF 2019-03-26 09:34:28+01:00 Preparing for fork choice tid=4794 connectedPeers=0 node=4
DBG 2019-03-26 09:34:29+01:00 Replaying state transitions tid=4794 ancestors=1 prevStateSlot=0 stateRoot=246EF52C stateSlot=15 node=4
INF 2019-03-26 09:34:29+01:00 Storing state tid=4794 stateRoot=246EF52C stateSlot=0 node=4
DBG 2019-03-26 09:34:29+01:00 No head update this time tid=4794 headBlockRoot=DB12D744 headBlockSlot=12 node=4
NOT 2019-03-26 09:34:29+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=4794 expectedEpoch=4 expectedSlot=38 fromNow=3 headSlot=12 node=4
INF 2019-03-26 09:34:29+01:00 Scheduling next slot start action block tid=4794 fromNow=3 slot=39 node=4
DBG 2019-03-26 09:34:29+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4794 remote=Node[127.0.0.1:50006] node=4
INF 2019-03-26 09:34:29+01:00 GossipSub Peer disconnected tid=4794 peer=Node[127.0.0.1:50000] reason=BreachOfProtocol node=4
Traceback (most recent call last)
beacon_node.nim(785) beacon_node
beacon_node.nim(704) start
beacon_node.nim(676) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
asyncmacro2.nim(36) handshakeImpl_continue
rlpx.nim(1115) handshakeImplIter
asyncmacro2.nim(306) disconnectAndRaise
asyncmacro2.nim(36) disconnectAndRaise_continue
rlpx.nim(51) disconnectAndRaiseIter
asyncmacro2.nim(306) disconnect
asyncmacro2.nim(36) disconnect_continue
rlpx.nim(1139) disconnectIter
rlpx.nim(1102) callDisconnectHandlers
asyncmacro2.nim(306) GossipSubPeerDisconnect
asyncmacro2.nim(36) GossipSubPeerDisconnect_continue
gossipsub_protocol.nim(41) GossipSubPeerDisconnectIter
DBG 2019-03-26 09:34:29+01:00 Exception in rlpxConnect topics="rlpx" tid=4794 err="Future still in progress.\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) handshakeImpl_continue\n rlpx.nim(1119) handshakeImplIter\n asyncfutures2.nim(329) read\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) BeaconSyncHandshake_continue\n sync_protocol.nim(87) BeaconSyncHandshakeIter\n asyncfutures2.nim(324) read\n ]#\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) postHelloSteps_continue\n rlpx.nim(1208) postHelloStepsIter\n asyncfutures2.nim(324) read\n ]#\nException message: Future still in progress.\nException type:" exc=ValueError remote=Node[127.0.0.1:50000] node=4
DBG 2019-03-26 09:34:29+01:00 Unexpected neighbours, probably came too late topics="discovery" tid=4613 remote=Node[127.0.0.1:50004] node=0
ERR 2019-03-26 09:34:29+01:00 dispatchMessages failed topics="rlpx" tid=4794 err="Data incomplete!\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) readExactly_continue\n stream.nim(1284) readExactlyIter\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) recvMsg_continue\n rlpx.nim(404) recvMsgIter\n asyncfutures2.nim(324) read\n ]#\nException message: Data incomplete!\nException type:" node=4
ERR 2019-03-26 09:34:29+01:00 dispatchMessages failed topics="rlpx" tid=4613 err="Data incomplete!\nAsync traceback:\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) readExactly_continue\n stream.nim(1284) readExactlyIter\n #[\n beacon_node.nim(785) beacon_node\n beacon_node.nim(704) start\n beacon_node.nim(676) run\n asyncloop.nim(808) runForever\n asyncloop.nim(260) poll\n asyncmacro2.nim(36) recvMsg_continue\n rlpx.nim(404) recvMsgIter\n asyncfutures2.nim(324) read\n ]#\nException message: Data incomplete!\nException type:" node=0
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice tid=4794 connectedPeers=1 node=4
INF 2019-03-26 09:34:33+01:00 Scheduling next slot start action block tid=4779 fromNow=5 slot=40 node=8
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice tid=4788 connectedPeers=1 node=2
INF 2019-03-26 09:34:33+01:00 Preparing for fork choice tid=4779 connectedPeers=1 node=8
Note: I'm looking at Nim for the first time so this might just be incorrect.
The following defines a BLSPubKey
which is used in the ValidatorRecord
type definition
https://github.com/status-im/nim-beacon-chain/blob/ca4082b316689c8702dc8ba665611de31e3a92cf/beacon_chain/datatypes.nim#L63-L65
I believe that when you serialize BLSPubKey
with SSZ that you will get a length prefix in the byte output. Because the length is known, we will likely use int384
or bytes48
. Both of these types will not require a length prefix when serialized with SSZ.
EDIT
It looks like the items in this aggregateSig
will suffer the same fate.
https://github.com/status-im/nim-beacon-chain/blob/ca4082b316689c8702dc8ba665611de31e3a92cf/beacon_chain/datatypes.nim#L77
Note, that while using the length of the BLSPubKey
works for the bn256 curve for aggregate sig, if we use BLS12-381, the signatures are not the same length as the pub keys. One is 48bytes while the other is 96bytes (depending on if you G1 or G2 for the key).
This part of the test suite is very slow and takes several minutes on Windows with beacon-chain built for 32-bit:
[OK] Passes through epoch update, empty block
These errors are being thrown by the straggler node that's run with run_node.sh 9
after the other 9 nodes are already running for 30 or so minutes. They happen rarely and are not easy to reliably reproduce - in fact, they do not happen again once the node is in sync with others.
This happens on Ubuntu 18.04.2.
INF 2019-03-12 15:52:12+01:00 Forward sync imported blocks tid=22243 badBlocks=0 bodies=455 goodBlocks=455 headers=455 roots=455 node=9
DBG 2019-03-12 15:52:12+01:00 Exception in rlpxConnect topics="rlpx" tid=22243 err="Can\'t obtain a value from a `none`\nAsync traceback:\n beacon_node.nim(627) beacon_node\n asyncloop.nim(761) waitFor\n asyncloop.nim(238) poll\n asyncmacro2.nim(36) BeaconSyncHandshake_continue\n sync_protocol.nim(108) BeaconSyncHandshakeIter\n options.nim(132) get\n #[\n beacon_node.nim(627) beacon_node\n asyncloop.nim(761) waitFor\n asyncloop.nim(238) poll\n asyncmacro2.nim(36) postHelloSteps_continue\n rlpx.nim(1206) postHelloStepsIter\n asyncfutures2.nim(325) read\n ]#\nException message: Can\'t obtain a value from a `none`\nException type:" exc=UnpackError:ObjectType remote=Node[127.0.0.1:50001] node=9
INF 2019-03-12 15:52:12+01:00 Syncing state from remote peers tid=22243 finalized_epoch=0 target_slot_epoch=72 node=9
INF 2019-03-12 15:52:12+01:00 Starting beacon node tid=22243 SECONDS_PER_SLOT=6 SHARD_COUNT=4 SLOTS_PER_EPOCH=8 SPEC_VERSION=0.4.0 slotsSinceFinalization=583 stateSlot=50 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=90 pubKey=14aa2583a999066ec6caa72a3fc19e80d8936f6856d447dd043aa9b126aa63bcaac876266d80913071777984d8d30563 validator=14aa2583 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=91 pubKey=0eb8b1b309a726fa5af6a6228385214a48788a1f23fe03cd46e16e200ed7d8909394d2e0b442ef71e519215765ca6625 validator=0eb8b1b3 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=92 pubKey=8c7b0e11f9bc3f48d84013ef8e8575aeb764bc1b9bf15938d19eb191201011365c2b14d78139a0f27327cb21c1b8bf3d validator=8c7b0e11 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=93 pubKey=0d08a52857017fd5cab3a821ccb8f5908c96cf63c5a5647209c037e2ea1c56f9650ec030b82ffdce76d37672d942e45b validator=0d08a528 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=94 pubKey=88f5540a9977fd2ee7dea836ed3dafa5d0b1fc9c5d5f1689e91ec49cdef989976c51502c3764025ef8ff542ef3b170ea validator=88f5540a node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=95 pubKey=8ff7cc69f007f11481c91c6f9b20698998a0c2e9a2928bec8eea7507c7ad73a9d1d218cfdb279c4d2132d7da6c9e513e validator=8ff7cc69 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=96 pubKey=0fb72b4c111da98379f195da4e5c18462acc7ece85cd66894fbaf69ddab3d3bb0b6957ea0042b7705937919189e6a531 validator=0fb72b4c node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=97 pubKey=812b2d0546aa77dec2d55406b0131ed580c079c1aeb76eb2ca076b7b58289fa9d781069a2e11fe2199f1e02c5dd70e6a validator=812b2d05 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=98 pubKey=8a10e1055b14a89cc3261699524998732fddc4f30c76c1057eb83732a01416643eb015a932e4080c86f42e485973d240 validator=8a10e105 node=9
DBG 2019-03-12 15:52:12+01:00 Attaching validator tid=22243 idx=99 pubKey=029e520a73ec28f4e2e45050c93080eeaee57af1108e659d740897c3ced76ceb75d106cb00d7ed25ec221874bf4b235a validator=029e520a node=9
INF 2019-03-12 15:52:12+01:00 Local validators attached tid=22243 count=10 node=9
DBG 2019-03-12 15:52:12+01:00 Scheduling epoch actions tid=22243 epoch=72 stateEpoch=6 node=9
NOT 2019-03-12 15:52:13+01:00 Replaying state transitions tid=22243 ancestors=3 prevStateSlot=552 stateRoot=02893E03 stateSlot=50 node=9
INF 2019-03-12 15:52:13+01:00 Updated head tid=22243 headBlockRoot=7A7528B2 stateRoot=22ABAD68 stateSlot=554 node=9
NOT 2019-03-12 15:52:13+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=22243 expectedEpoch=72 expectedSlot=583 fromNow=2 stateSlot=554 node=9
INF 2019-03-12 15:52:13+01:00 Scheduling next slot start action block tid=22243 fromNow=2 slot=584 node=9
INF 2019-03-12 15:52:16+01:00 Scheduling next slot start action block tid=22243 fromNow=5 slot=585 node=9
DBG 2019-03-12 15:52:16+01:00 Scheduling epoch actions tid=22243 epoch=73 stateEpoch=69 node=9
DBG 2019-03-12 15:52:16+01:00 No new head found tid=22243 blockRoot=7A7528B2 stateRoot=22ABAD68 stateSlot=554 node=9
NOT 2019-03-12 15:52:16+01:00 Delaying epoch scheduling, head too old - scheduling new attempt tid=22243 expectedEpoch=73 expectedSlot=584 fromNow=5 stateSlot=554 node=9
DBG 2019-03-12 15:52:16+01:00 Attestation received tid=22243 beaconBlockRoot=7E0738DF justifiedBlockRoot=FA19A839 justifiedEpoch=70 shard=0 signature=0cb50812 slot=584 node=9
ERR 2019-03-12 15:52:16+01:00 dispatchMessages failed topics="rlpx" tid=22243 err="/home/swader/repos/nim-beacon-chain/beacon_chain/spec/validator.nim(153, 12) `epoch <= next_epoch` Previous epoch: 68, epoch: 73, Next epoch: 70\nAsync traceback:\n beacon_node.nim(641) beacon_node\n beacon_node.nim(595) run\n asyncloop.nim(756) runForever\n asyncloop.nim(238) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(519) dispatchMessagesIter\n rlpx.nim(266) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(36) emit_thunk_continue\n rlpx.nim(767) emit_thunkIter\n asyncmacro2.nim(306) emit\n asyncmacro2.nim(36) emit_continue\n gossipsub_protocol.nim(61) emitIter\n serialization.nim(65) :anonymous\n beacon_node.nim(585) :anonymous\n beacon_node.nim(532) onAttestation\n attestation_pool.nim(246) add\n attestation_pool.nim(139) validate\n validator.nim(153) get_crosslink_committees_at_slot\n system.nim(3790) failedAssertImpl\n system.nim(3783) raiseAssert\n system.nim(2830) sysFatal\n #[\n beacon_node.nim(641) beacon_node\n beacon_node.nim(595) run\n asyncloop.nim(756) runForever\n asyncloop.nim(238) poll\n asyncmacro2.nim(36) dispatchMessages_continue\n rlpx.nim(519) dispatchMessagesIter\n rlpx.nim(266) invokeThunk\n asyncmacro2.nim(306) emit_thunk\n asyncmacro2.nim(39) emit_thunk_continue\n rlpx.nim(767) emit_thunkIter\n asyncfutures2.nim(325) read\n ]#\nException message: /home/swader/repos/nim-beacon-chain/beacon_chain/spec/validator.nim(153, 12) `epoch <= next_epoch` Previous epoch: 68, epoch: 73, Next epoch: 70\nException type:" node=9
INF 2019-03-12 15:52:16+01:00 GossipSub Peer disconnected tid=22243 peer=Node[127.0.0.1:50000] reason=ClientQuitting node=9
Errors are in second and second-to-last line.
The node does not crash in this instance, it continues along after these errors are thrown.
beaconnode createChain crashes with a strange error related to Cygwin in the current master https://github.com/status-im/nim-beacon-chain/tree/dda24619e824ef78c216d8a99ee280e57f182aa8 compared to the master from a week ago: https://github.com/status-im/nim-beacon-chain/tree/fix-sim-init-windows.
Full backtrace with debugging symbol:
#0 __chkstk_ms () at ../../../../../src/gcc-4.9.1/libgcc/config/i386/cygwin.S:172
#1 0x0061b2aa in get_initial_beacon_state_vi9cRmmSUyyCGR9cJE84pqSw (initial_validator_deposits=0x5f2c3165, initial_validator_depositsLen_0=1887007839, genesis_time=6857984371532455724, latest_eth1_data=0x14cbaa8, flags=116 't', Result=0x14cbae8) at D:/Devel/Status/nim-beacon-chain/beacon_chain/spec/beaconstate.nim:159
#2 0x004a7f48 in createStateSnapshot_ahCWD5wcxfkrgWQSdv9cjfQ (startup=..., outFile=0x173a8e8) at D:/Devel/Status/nim-beacon-chain/beacon_chain/trusted_state_snapshots.nim:33
#3 0x00439e9d in NimMainModule () at D:/Devel/Status/nim-beacon-chain/beacon_chain/beacon_node.nim:402
#4 0x00439c11 in NimMainInner () at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3404
#5 0x00439c35 in NimMain () at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3412
#6 0x00439c5f in main (argc=4, args=0x1b914a8, env=0x1b91d98) at C:/Users/Beta/.choosenim/toolchains/nim-0.19.2/lib/system.nim:3419
Note that this is with 32-bit Nim
As of f9e0418, the local sim is relatively stable, but node fail occasionally with the following error:
WRN 2019-03-28 10:52:13+02:00 Unexpected crosslink shard tid=22764 acceptable_crosslink_data="@[(epoch: 536870914, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000), (epoch: 536870916, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)]" actual_epoch=4 attestation_data_previous_crosslink="(epoch: 536870914, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)" crosslink_data_root=0000000000000000000000000000000000000000000000000000000000000000 epoch=4 state_latest_crosslinks_attestation_data_shard="(epoch: 536870915, crosslink_data_root: 0000000000000000000000000000000000000000000000000000000000000000)" node=8
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(36) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncmacro2.nim(306) handleProposal
asyncmacro2.nim(36) handleProposal_continue
beacon_node.nim(511) handleProposalIter
asyncmacro2.nim(306) proposeBlock
asyncmacro2.nim(36) proposeBlock_continue
beacon_node.nim(359) proposeBlockIter
system.nim(3790) failedAssertImpl
system.nim(3783) raiseAssert
system.nim(2830) sysFatal
[[reraised from:
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(36) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncmacro2.nim(306) handleProposal
asyncmacro2.nim(39) handleProposal_continue
beacon_node.nim(511) handleProposalIter
asyncfutures2.nim(324) read
]]
[[reraised from:
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(39) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncfutures2.nim(324) read
]]
[[reraised from:
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
asyncfutures2.nim(369) cb
]]
Error: unhandled exception: /home/zahary/nimbus/vendor/nim-beacon-chain/beacon_chain/beacon_node.nim(359, 12) `ok`
Async traceback:
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(36) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncmacro2.nim(306) handleProposal
asyncmacro2.nim(36) handleProposal_continue
beacon_node.nim(511) handleProposalIter
asyncmacro2.nim(306) proposeBlock
asyncmacro2.nim(36) proposeBlock_continue
beacon_node.nim(359) proposeBlockIter
system.nim(3790) failedAssertImpl
system.nim(3783) raiseAssert
system.nim(2830) sysFatal
#[
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(36) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncmacro2.nim(306) handleProposal
asyncmacro2.nim(39) handleProposal_continue
beacon_node.nim(511) handleProposalIter
asyncfutures2.nim(324) read
]#
#[
beacon_node.nim(809) beacon_node
beacon_node.nim(734) start
beacon_node.nim(707) run
asyncloop.nim(808) runForever
asyncloop.nim(260) poll
beacon_node.nim(658) :anonymous
asyncmacro2.nim(306) onSlotStart
asyncmacro2.nim(39) onSlotStart_continue
beacon_node.nim(618) onSlotStartIter
asyncfutures2.nim(324) read
]#
Exception message: /home/zahary/nimbus/vendor/nim-beacon-chain/beacon_chain/beacon_node.nim(359, 12) `ok`
Exception type: [AssertionError]
Upstream spec change see ethereum/beacon_chain#60
there have been a lot of issues regarding which combinations of state.slot
and passed-in slot are safe for committee selection - write a test case to highlight what works
beacon_node.nim(779) beacon_node
beacon_node.nim(673) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncmacro2.nim(36) sendMsg_continue
rlpx.nim(285) sendMsgIter
asyncfutures2.nim(324) read
[[reraised from:
beacon_node.nim(779) beacon_node
beacon_node.nim(673) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncmacro2.nim(39) sendMsg_continue
excpt.nim(145) sendMsgIter
]]
[[reraised from:
beacon_node.nim(779) beacon_node
beacon_node.nim(673) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncfutures2.nim(369) cb
]]
Error: unhandled exception: (32) Broken pipe
Async traceback:
beacon_node.nim(779) beacon_node
beacon_node.nim(673) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncmacro2.nim(36) sendMsg_continue
rlpx.nim(285) sendMsgIter
asyncfutures2.nim(324) read
#[
beacon_node.nim(779) beacon_node
beacon_node.nim(673) run
asyncloop.nim(756) runForever
asyncloop.nim(238) poll
asyncmacro2.nim(39) sendMsg_continue
excpt.nim(145) sendMsgIter
]#
Exception message: (32) Broken pipe
Exception type: [Exception]
The simulation can (always ?) get stuck in a state slot as of 2d52d5c
Wrote /home/cheatfate/nim-beacon-chain/tests/simulation/data/startup.json
Wrote /home/cheatfate/nim-beacon-chain/tests/simulation/data/state_snapshot.json
INF 2019-02-14 06:45:49-06:00 Initializing DB tid=23528
beacon_node.nim(427) beacon_node
beacon_node.nim(61) init
beacon_chain_db.nim(61) persistState
beacon_chain_db.nim(45) lastFinalizedState
Error: unhandled exception: Internal error: Database has no finalized state [Exception]
See alternative implementation in Lighthouse (Rust):
And related issue in Prysm (Go):
Discussing with @djrtwo, for testing purposes beacon chains will have a couple of different configurations of the "constants" in the spec.
For example SLOTS_PER_EPOCH
is 8 here: https://github.com/ethereum/eth2.0-tests/blob/0d4bb97d71e3b7e983fa5c0cb22039e2aba26b1a/state/sanity-check_small-config_32-vals.yaml#L27
This would need rewriting lots of the code from array to sequences. Also it will prevents us many optimizations like preallocating a buffer or avoiding GC memory.
This is the easier solution has only a restricted amount of configs are needed.
Example:
const constantsConfig {.strdefine.} = "official"
when constantsConfig == "official":
import constants/official
elif constantsConfig == "test1":
import constants/test1
and compile the relevant tests with nim c -d:constantsConfig=test1 beacon_node
This is the most flexible but quite involved (overkill?) as it would require a compile-time yaml parser and then a macro that generates the proper constants during compilation.
so regarding logging, to tune it so that operators can run INF
and developers DBG
, I would suggest the following guidelines:
NOT
noteworthy events that are unusual as far as normal operation goes: (deep?) chain reorgs, slashings etcINF
logs anything that is sufficient to recreate the state along important checkpoints in the application, but does so in a minimal way, so as not to overwhelm log collectors - minimal in this sense means that it strives to avoid duplicating the same information - kind of like a normalized database - people running the app on a server should not be discouraged from always keeping INF onDBG
denormalizes the above and adds more checkpoints - repetition for convenience is ok, but we still keep it out of tight loops - developers will have this on by defaultTRC
anything goeswe can stick these in a document somewhere once we have a bit more agreement on the practical application of them, in nimbus specifically
After the fixing of #85 and #89, we now have a new issue during epoch transition.
Reproduced also on @yglukhov side:
With added logs on epochs and slots:
- in scheduleEpochActions
-- https://github.com/status-im/nim-beacon-chain/blob/2794181889801c17f6d9ecca180e04c9bada29f7/beacon_chain/beacon_node.nim#L275-L310
- in get_crosslink_committees_at_slot
-- https://github.com/status-im/nim-beacon-chain/blob/2794181889801c17f6d9ecca180e04c9bada29f7/beacon_chain/spec/validator.nim#L69-L80
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:24:56+01:00 Waiting for connections thread=0 node=0
INF 2019-02-07 19:24:56+01:00 RLPx listener up thread=0 self=enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000 node=0
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=3
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=2
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://18ceedebc1034b2d97fe1a3dd20f4cb1de7e740b0dd59fd80425cfeec2e2f07b07639637294e5d406f71f2fcbf91160fa09637ca30f4a4f063628bf484ac08a0@127.0.0.1:50002 node=2
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=2
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://14b6f6abf2de6256e82fc5961cdd545cef2d1800d5d3d5c6fd311d85deccfb2ddb45b1e4cfb2dda31edb08634fddd4535176c5c2024d0b14866b4ae7e6ba1f44@127.0.0.1:50003 node=3
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=3
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=1
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
fork_slot: 9223372036854775808
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=6
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://252ba14c6ee47dfdf1c8931b8f47e2265ad257ec8d9430103ed7f9b6b14f687a1a816202256d147641c3059541a294b9efaeed2995646264ffc121eea5304325@127.0.0.1:50001 node=1
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=1
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=5
fork_slot: 9223372036854775808
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://56bde0fab36731a9b9920cd9ecbc66cf0b06f0a5096aac4dc16c60c8e3b4c15cf067ef92edaa91172f376acfce309368a41a9fdb3e89f458199112ceffa27536@127.0.0.1:50006 node=6
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=6
node slot: 0
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://dfe063548e8ac825d60cd0cd367273ad99c0c9547b800609ee5f6951ca0027167b8de026e16ab1b2d3c2964936df0649fff7d7e4502a34925cf44d5b58c1df3c@127.0.0.1:50005 node=5
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=5
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=9
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=7
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=4
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://bbe000c0a94aecd030402128b829bd12562e1bf4d8300c80a2769eb2b30eabc98a799d279cfce207a04360e139cc7858f32d7eaa53705320d15712c35834833c@127.0.0.1:50009 node=9
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=9
node slot: 0
node finalized epoch: 0
INF 2019-02-07 19:25:02+01:00 Connecting to bootstrap nodes thread=0 bootstrapNodes=@[enode://e1f94e6fae39ffabcd961b4a31b0a72e1933c174fa5bbd4046a1798a66babb34d7fee1566092e46069537efa8ef7994ffec66b1787204a9a6a73b9ba1dbcb31a@127.0.0.1:50000] node=8
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://cdf6260a7ebdebd0e498088afebe8adf3201586db86ccf7668574ea21764fc33f1141153c869671808971ac3a44a13460bcbc5e9539c7a9f67ae7b5ce323bd78@127.0.0.1:50007 node=7
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=7
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://eb5cf6725dd129edb77f25405ae538183303ddce9155839a21c64fede1577ed4733f1538b2862ce97b6153220c07232c9d4181dada8174156bde151dbced6085@127.0.0.1:50008 node=8
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=8
INF 2019-02-07 19:25:02+01:00 RLPx listener up thread=0 self=enode://1a6c56c1cf9cb7bbfe3ddb18906876a7ee314958b745b1d0fefbd747013a94d90d9a0db1392b38f868627b9d97763acf67c6d7c96d2ea02d9569f579d5d7401e@127.0.0.1:50004 node=4
INF 2019-02-07 19:25:02+01:00 Bonding to peer topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=4
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=3
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=3
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=2
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=2
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=3
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=2
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=1
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=1
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=1
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=6
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=6
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=5
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=5
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=6
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=5
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=9
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=9
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=7
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=7
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=9
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=7
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=4
DBG 2019-02-07 19:25:03+01:00 Bonding failed, didn't receive pong from topics="discovery" thread=0 n=Node[127.0.0.1:50000] node=8
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=4
INF 2019-02-07 19:25:03+01:00 Failed to bond with bootstrap nodes topics="discovery" thread=0 node=8
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=8
DBG 2019-02-07 19:25:03+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=4
INF 2019-02-07 19:25:03+01:00 Starting beacon node thread=0 slotsSinceFinalization=-144115188075855872 stateSlot=9223372036854775808 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=0 pubKey=1547bfa50ff2dbcd3c6df014657310cbbd7262c9dcfd62875b085dfa947a20bf0bec359a64c2d41c06f8b61d1ad9ffdf validator=1547bfa5 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=1 pubKey=17f186031dede4d908aef7882445328bd98a390855dfce07fc12be73b90c77ddb0fdedd003ff23a1a101e6e842b07ef3 validator=17f18603 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=2 pubKey=8249e0535a6f3cfd206777e4057edbb83a927067c77955283663f105e44258fcdbf0e55c436551489df5ab1e20c9ef55 validator=8249e053 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=3 pubKey=0ca6764168121779715ee331e867f9da0c1d3b889b8d1a17abee3272b3c362933f3af2eba47dd8cac289a2f71bdeb4dd validator=0ca67641 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=4 pubKey=027fa75ec2ba5a3dc7b1ed314de771c7f08135e8d4c1cc6bb2d696669e7be4627fa570be348f38356596d5c5798bc874 validator=027fa75e node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=5 pubKey=988c71630e0722245655a2ba50269c3b95a4c2be3468bb577f2a21633c4bee361fbb00a03ce690086000ac1bf197a5bb validator=988c7163 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=6 pubKey=0a42564e2ac7bae576e62304a524f8a372b524f015543801c13d45fb8967f5da505a9cde19fbbeee87c3f49e88faf4a3 validator=0a42564e node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=7 pubKey=16efb9733a20a511fa4e1d4ed06a8594496a8424df7d5dae31e4ac14d87a88119a4382a26af8636375430b3525cd86db validator=16efb973 node=0
DBG 2019-02-07 19:25:03+01:00 Attaching validator thread=0 idx=8 pubKey=010c62895e4194ec694033016128347c63946eb7b15a2cf5195c0260a02663435de8b8e9aeaf6f802ffc2bb11d073798 validator=010c6289 node=0
INF 2019-02-07 19:25:03+01:00 Local validators attached thread=0 count=9 node=0
DBG 2019-02-07 19:25:03+01:00 Genesis epoch thread=0 GENESIS_EPOCH=144115188075855872 node=0
DBG 2019-02-07 19:25:03+01:00 Scheduling epoch actions since genesis thread=0 epoch=144115188075855872 node=0
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
i: 0
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
i: 1
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
i: 2
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
i: 3
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
[...]
i: 22
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
INF 2019-02-07 19:25:03+01:00 Scheduling block proposal thread=0 fromNow=4 slot=9223372036854775832 validator=027fa75e node=0
i: 23
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
[...]
i: 42
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
INF 2019-02-07 19:25:03+01:00 Scheduling block proposal thread=0 fromNow=124 slot=9223372036854775852 validator=1547bfa5 node=0
i: 43
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
[...]
i: 49
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
INF 2019-02-07 19:25:03+01:00 Scheduling block proposal thread=0 fromNow=166 slot=9223372036854775859 validator=988c7163 node=0
i: 50
[...]
i: 56
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
INF 2019-02-07 19:25:03+01:00 Scheduling block proposal thread=0 fromNow=208 slot=9223372036854775866 validator=16efb973 node=0
i: 57
[...]
i: 61
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
i: 62
Input epoch: 144115188075855872
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
Input epoch: 144115188075855873
Previous epoch: 144115188075855872
Current epoch: 144115188075855873
Next epoch: 144115188075855874
i: 63
Input epoch: 144115188075855873
Previous epoch: 144115188075855872
Current epoch: 144115188075855872
Next epoch: 144115188075855873
beacon_node.nim(438) beacon_node
beacon_node.nim(397) processBlocks
beacon_node.nim(306) scheduleEpochActions
validator.nim(86) get_crosslink_committees_at_slot
system.nim(3879) failedAssertImpl
system.nim(3872) raiseAssert
system.nim(2918) sysFatal
Error: unhandled exception: /Users/<PATH>/nim-beacon-chain/beacon_chain/spec/validator.nim(86, 10) `epoch < next_epoch` [AssertionError]
DBG 2019-02-07 19:25:05+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=5
DBG 2019-02-07 19:25:05+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=9
DBG 2019-02-07 19:25:05+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=6
DBG 2019-02-07 19:25:05+01:00 Not enough nodes topics="discovery" thread=0 present=0 requested=1 node=1
(On MacOS)
(using the code on -master
)
$ nimble build
$ nimble test
both work/pass.
But when running:
$ ./tests/simulation/start.sh
I get the following error:
could not load: librocksdb(|_lite).dylib
Do I need to manually install librocksdb
?
We have many issues related to epochs and slots and their unique treatment in the specs:
We probably need 4 distinct types. 2 for each epoch and slots.
type
Slot = distinct uint64
Epoch = distinct uint64
PrintableSlot = distinct string
PrintableEpoch = distinct string
With Slot and Epoch having no conversion to string defined and PrintableSlot/Epoch having only conversion to string defined.
Looks like an off-by-one somewhere, epoch numbers don't match:
WRN 2019-03-04 14:51:43-06:00 Unexpected crosslink shard tid=18654 attestation_data_latest_crosslink="(epoch: 1152921504606846992, shard_block_root: 0000000000000000000000000000000000000000000000000000000000000000)" epoch=18 shard_block_root=0000000000000000000000000000000000000000000000000000000000000000 state_latest_crosslinks_attestation_data_shard="(epoch: 1152921504606846993, shard_block_root: 0000000000000000000000000000000000000000000000000000000000000000)" node=5
our code confuses the naming here vs spec with regards to zero padding - needs updating (hash_tree_root
is the padded version in spec)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.