salesforce / apollo Goto Github PK

Implement a windowed transaction deduplication. Transactions that have successfully been embedded on a block will be tracked for a given window. Upon block creation, any duplicated transactions are filtered from the block, new transaction digests are added to the dedup window.

Add mechanism to resubmit txns after timeout without regeneration, as the nonce in the transaction makes each txn generation unique.

The HEX-BLOOM scheme provides an alternative to the MerkleTree and cheesy current mechanism for recording and authenticating CHOAM checkpoint state blocks. HEX-BLOOM is secure and efficient in both time and space and makes an excellent fit for this purpose.

Full scenario, Gorgoneion bootstrap to Delphinius ACL testing

To offload heap memory required, store the SignedNotes, Accusations, Joins, Observations using MVStore. This can then be configured for disk or off heap storage for scalability.

Now that we have Virtual Threads, we can optionally remove timeout on data source generation and simply block waiting for the unit's data. This is required in the Genesis use of Ethereal as we're blocking on 2/3 of the genesis committee to construct the genesis block. If the DS times out, we miss slow joins.

For Ghost, we expect that the set sizes of the members to be large and ponderous. And hopefully long lived. This means that any differences between any given replication partner will be small, even though the cardinality of the sets is ginormous (billions).

The current Distributed Bloom Filter set replication mechanism seems wildly inappropriate for this particular context, and I believe the schemes using counting bloom filters, invertible bloom filters and estimators - perhaps min wise sketches, etc - may need to be used at scale. For now, we just use what we got.

Use the block height of the SQL state as the Zookie equivalent for Delphinius. This provides the same "happened before" semantics that the Atomic clock (and additional time protocols) ensure.

I believe the Mutator abstraction in sql-state can be turned into a JDBC mock with the same effect. Will see if I can use the reactive (non blocking) form of JDBC so's I can nicely accommodate the async nature of the sql state machine

https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b

Add ability to derive encryption key pair from signature key pair

Transactions can truly timeout due to the underlying system behavior. Add convent pattern for retrying idempotent transactions in Delphinius.

Include the ULong sequence number of the signing KERL establishment event coordinate in the JohnHancock. Use double dispatch to produce the corresponding private keys as well as simple lookup for an alternative pattern.

No reason not to use the JOOQ DSL in addition to the string based API of the CHOAM mutator.

provide additional authnz using Fernet tokens

Augment Fireflies to provide step wise, quantum changes in membership of a view. Simplified view membership event interaction with coupled components.

Provide access to the SQL state machine current block to the stored procedures

Analysis of checkpoint bootstrapping flappers has shown that it's easily possible for a bootstrapping node to start bootstrapping on the most recent checkpoint A (from group POV) and in the process of recovering another checkpoint B will occur in the system. This wipes out the blocks between checkpoints A and B, and the current Boostrapper fails to recover - it simply keeps on trying to synchronize these blocks which no longer exists.

See http://www.vldb.org/pvldb/vol12/p1539-nathan.pdf

The current implementation is fine, but wastes a byte per cell. There is a very promising, very simple implementation using:

A Multi-partitioning Approach to Building Fast and Accurate Counting Bloom Filters