Bug: Prysm broadcasts blocks with invalid attestations (Antithesis Experiment) about prysm

[parithosh]

Also note that there was an extremely deep reorg (depth 28) seen on this network. There shouldn't be any straightforward case in which a reorg of such depth happens on a small network that close to genesis - so could be some other factors involved as well.

[nisdas]

Hey, I posted this offline too but for clarity will post the same answer here:

On why we broadcasted and then process an invalid block, this is how our workflow is:

• Build block and the immediately broadcast it
• Process the block locally after

So it fails on the second step, the reason we do it is to allow blocks to be propagated as fast as possible across the network

We have seen this issue before, it happens due to deep reorgs and attestations from a different shuffling are in the same pool. Which is what causes the invalid blocks to be produced, there isn't an easy solution to it yet because it would involve deep changes to the attestation pool that we are vary of.

A solution to this would be to make our attestation pool re-org aware with respect to changed shufflings and just purge attestations from the different(old) branch. On whether this would be a good idea maybe @terencechain @potuz might have more thoughts on it

[qu0b]

@parithosh The chain reorgs are both after the chain managed to finalize:

last slot with invalid attestations: 116

First reorg is from slot 143 -> 124
Second is from 150 -> 137

[  1804.570738] [system;capturefs;files;/service_prysm-nethermind-0--prysm-bn]        [I] time="2023-11-17 12:30:09" level=info msg="Chain reorg occurred" commonAncestorRoot=0x81ba77b368441be27d63f495eb42242fa22bb70faa3f08f6d4f044df03a542fe depth=28 distance=37 newRoot=0x3373e44863ea4045b8b8c8710cb3f238e11119650cf839566d96ce2f71c0481e newSlot=124 newWeight=0 oldRoot=0xa836cb36e853cbeae02bae40768e2e3b802db5630fa20710438321eee7cc4383 oldSlot=143 oldWeight=0 prefix=blockchain
[  1850.807387] [system;capturefs;files;/service_prysm-geth-0--prysm-bn]              [I] time="2023-11-17 12:30:56" level=info msg="Chain reorg occurred" commonAncestorRoot=0x8a8574988b86c24cc8eb3c2b82ce84b22077c21b2ee058e6ff86f19f65a8fa30 depth=15 distance=17 newRoot=0x4a2035133aafbab2a4139864a414c89163f71aa7b09bf35fd5ae14a8d4d7a497 newSlot=137 newWeight=0 oldRoot=0x9088c8169555338677bdada4ad9735e7f33ff7d75e95552105b40d9e634da871 oldSlot=150 oldWeight=0 prefix=blockchain

We also wait with faults to the network until around slot 120.

[parithosh]

I tried reproducing the issue elsewhere and unfortunately wasn't able to :/

But @nisdas if the reorg happens after the invalid sigs were seen, then they're likely unrelated right?