anoma / ferveo Goto Github PK

Could add support for enclaves (when they exist at the deployment environment) for:

• sealing secret shares
• shares generation
• decryption operations

Details TBD.

Relevant refs:

• Targetting SGX from Rust: https://github.com/fortanix/rust-sgx
• in AWS: https://aws.amazon.com/ec2/nitro/nitro-enclaves/
• in Azure: https://docs.microsoft.com/en-us/azure/confidential-computing/enclave-aware-containers

Loop in https://github.com/anoma/ferveo/blob/master/tpke/src/decryption.rs#L67 can be sped up by using arkworks VariableBaseMSM.

The code below shows an initial approach (currently failing with errors):

        use ark_ec::msm::VariableBaseMSM;
        // sum_D_j = { [\sum_j \alpha_{i,j} ] D_i }
        for (D, alpha_j, sum_alpha_D_i) in izip!(shares.iter(), alpha_ij.iter(), sum_D_j.iter_mut())
        {
            let Dj = D.iter().map(|Dij| Dij.decryption_share).collect::<Vec<_>>();
            let alpha_j = alpha_j.iter().map(|a| a.into_repr()).collect::<Vec<_>>();
            *sum_alpha_D_i = VariableBaseMSM::multi_scalar_mul(&Dj, &alpha_j);
        }

There is a nontrivial amount of symmetric crypto in the protocol, almost all of it should be provided by existing crates.

This task involves:

1. Finalizing choice of symmetric crypto primitives (e.g. ChaCha20, BLAKE2b, hash-to-curve, etc)
2. Integrating final choice(s) into the codebase and the protocol documentation.

I think the high priority for symmetric crypto choices should be:

1. Consistency within the entire Anoma protocol
2. Widespread use in the broader crypto and Rust community
3. High performance considerations (e.g. Chacha20 for mobile, AES-NI, etc)

Caching $[\lambda*b] Z_{i,\omega_i}$ for some 2/3 weight subset of signers will benefit performance in the cases where the next block also has that same 2/3 weight subset.

Since Ferveo is intended to be an "online" protocol and some/many primitives being used are not constant-time/may have other side-channel vulnerabilities, there should be an analysis and potential mitigations investigated (as needed).

Fortunately Ferveo is not like TLS where the latency is highly important, so hopefully this should be straightforward.

Early idea for discussion.

For:

• t-limited byzantine adversary
• $\phi(x,y) = XAY$, $X$ vector of powers of $x$, $Y$ vector of powers of $y$, $A$ is the $(t+1) \times (t+1)$ binomial matrix
• univariate polynomials generated by fixing the value of $x$ ($\phi(i,y)$)

We could reduce the amount of operations in verify-poly and verify-point by generating a sparse matrix for the binomial's coefficients. E.g. a O(tlogt) dense matrix will reduce both verify-poly and verify-point to O(tlogt) exponentiations and O(tlogt) multiplications.

Additionally, we could use compression for the commitment. If we consider commitment compression, we need to be careful since it leaks information on which are the zero factors. Some initial thoughts on what to be careful about:

• last column of $A$ needs to have only non-zero coefficients, otherwise the secret will be recoverable by $t+1-k$ univariate polynomials, where $k$ the number of factors set to zero in last column
• $A_{0,0} \ne 0$ to maintain the degree of the binomial at $2t$ and of the univariates' at $t$

Setting aside performance, initially, the modified Haven VSS needs to be fully implemented feature wise. The three primary changes from the Haven paper are:

1. Using a weighted scheme, as described in the draft design proposal, aggregating all shares belonging to the same node into the same send/echo/ready messages.
2. Evaluating polynomials at omega^i instead of i. This can be done the simple way for now and FFT optimizations can be added later.
3. Implementing vector commitment as Merkle tree (or deciding that some other vector commitment scheme is superior)

Implementing the VSS to feature-completeness will allow the performance tweaking to happen in parallel to building the rest of the protocol.

• Part (I): Make sense of the code

  • What's the role of full nodes vs validators?
    • Even though a validator can check all steps in the DKG, the DKG is a subjective outcome, anyway. Because 2/3 can manipulate however they want, full nodes don't need to check the correctness of the DKG; they can just trust the DKG is valid.
    • The complicated part is not Ferveo related, but it's a question about how much work nodes need to do to be in sync with the network.
      • DKG is expensive computation
  • In src
    • dkg
      • Params
        • tau: Session identifier
        • security_threshold: Match Tendermint 2/3 threshold
      • State machine
        • Mirror what the state is on the blockchain
        • Everyone should have the same state
        • Since Tendermint has its own state machine, it's not clear if there's anything else to keep in mind
    • msg
      • For testing purposes
        • Serialization and message signing
          • Not what we'll use (presumably)
          • It's better to sign transactions inside Tendermint
      • There are some messages that happen outside of the ledger?
        • For the DKG, not everything has to be conceptually on the ledger but it's simpler if it is, i.e. if every node has access to the same data
        • It'd be nice if all data generated by the DKG can be deleted, as it is useless (even for the verifier)
        • Distinguish between gossip message vs message in the ledger
    • pvdkg
      • Handles single session
      • The structure PubliclyVerifiableDKG will handle all the context for handling DKG messages
      • impl PubliclyVerifiableDKG For constructing a new context it is required
        • Identity (ed_key)
          • It is important to know which messages came from me
            • Maybe the API would have to account for
            • Possibility: we remove the signing process outside the Ferveo mode
        • params
          • Everyone needs to agree on how many shares
        • pvss_params
          • TODO: Could be generated internally and remove it from the API
      • share
        • Only 2/3 highest weight validators call the share function
        • This is on the boundary of Tendermint and Ferveo
          • It's a proactive message, not in response to a previous message
          • It's is a validator responsibility to create such message
          • There is a liveness question about how to do this
            • It'd be easy to sort all validators by weight and call share
              • This is not good enough: Not all validators are live
                • There must be a timeout
                  • If times out, additional validators should be involved
            • Is there a cost of involving more than 2/3 validators from start?
              • Performance
              • This is the operation that generates the most data
                • If we want to allow everyone to share, we create 50% more data. It's overhead
              • Imposing a computing cost to verify
              • Conclusion: Everyone should call share!?
                • (Couldn't follow the reasoning)
    • aggregate
      • Once 2/3 threshold is reached, one of the validators need to call the aggregation function
        • It is not cheap, but not horribly expensive
        • It checks all messages and construct an aggregated shared message
        • The awkward bit is that the internal sharing messages don't get checked by anyone, they only get check when someone runs aggregate.
        • Once a block is finalized (reaches the 2/3 threshold)
    • find_by_key
      • Not an external (API) function
    • anounce
      • When validator announces they take part in the DKG
        • DKG should not have announce, it should be derived from Tendermint
          • Only use, getting the session key
          • Annoying because it required liveness on the part of the validator to post the session key. If not, there is no actual way to communicate the secret key shares to that validator. This is an awkward stage in the protocol.
      • We could have a weak liveness when someone has posted a previous session key
        • They can announce a refresh/update their session key, but it's optional
          • Then we don't have the risk of falling offline

• Open questions:

  • What the important API calls should look like coming from ABCI++?
  • Fears:
    • Compute load, latency, performance, etc.
      • It will drive iterations of the DKG

Is #24 up to date? We should also merge the rendered PDF with Git LFS.

The #14 implementation helped performance quite a bit, but there are so many subgroup checks that further improvement is needed.

Probably the best remaining trick is to batch many subgroup checks together. Maybe these PR:

arkworks-rs/algebra#127
arkworks-rs/algebra#130

Other relevant issues:

zcash/zcash#3470
Consensys/gnark-crypto#94
https://cseweb.ucsd.edu/~mihir/papers/batch.pdf
apache/incubator-milagro-crypto-rust#37

The fundamental problem is described very well in the DKG in the Wild paper: "the difficulty of differentiating between a slow node and a faulty node in the asynchronous setting are the primary sources of the added complexity."

Basically a slow node (or a faulty node that comes back online) must be able to complete the DKG, however, if the other nodes have already signed "ready", then it's difficult to guarantee share distribution for the slow/faulty node. The DKG in the Wild paper solves this because when HybridVSS succeeds, the slow nodes are guaranteed to recover because even if the dealer goes offline, everyone else can share to them.

I still think using univariate polynomials is a good idea because HybridVSS is just pretty expensive by itself. But we have to figure out exactly how to do it.

Just to enumerate some options:

1. Post everyone's shares in some kind of verifiable-but-encrypted form "on chain". Optimistically, if the shares are only 48 bytes and there are 10,000 shares, and say we need to run 50 VSS instances, that's about 2.4MB chain storage per DKG. Not sure if that estimate is really realistic though. The "verifiable" part might make it more expensive. Maybe the data amount is not prohibitively expensive, but not ideal either.
2. Keep trying to patch the univariate scheme to work cryptographically. I really think this should be possible, the challenge is in the details. Maybe there's suboptions for option (2), let's say (2a) and (2b). (2a) is to get consensus on which nodes complete each VSS (and overall DKG) instead of running each VSS separately - the way to make it work is maybe to not treat the VSS instances as independent (because then the faulty nodes might differ between VSS instances) but to also achieve consensus on what nodes have finished common sets of VSS. (2b) is to encrypt the shares in some publicly verifiable way (further investigation needed)
3. Maybe use cryptoeconomics instead of verification. That is, the dealer distributes a big blob of everyone's shares (encrypted individually) and each node has no hope of verifying that other's shares are valid, but if someone else's shares turn out invalid, this fact can be verifiably revealed later (even after VSS is complete) and the dealer can be slashed

I actually rather like option 3 now that I consider it more. It uses very little on chain data (more or less the protocol described in the call on Tuesday), has low cryptographic and computational complexity, and particularly cheap in the optimistic case where no one tries to cheat and there are few/no network failures.

The scalability of (1) bothers me. it wastes a lot of resources to handle a non-optimistic case (bad behavior by the dealer) and the scalability of number of shares is itself part of the security of the system (a larger number of shares and therefore increased granularity/resolution can help the security, particularly if there are many small weight nodes in the system)

Since we are assuming a consensus mechanism available, it appears we don't really need the leader change protocol with its added complexity. The fundamental goal is to make sure everyone agrees on the HybridVSS instances that are used to construct the key and the consensus mechanism can guarantee that. So, probably everyone should simulate being the leader in the HybridDKG protocol and use the consensus choice of HybridVSS results.

There are still some questions of how to do this exactly. The first possibility is keeping HybridDKG with an optimistic phase and a pessimistic phase. In the optimistic phase, every HybridVSS result has been shared with every node, so there is no need for a consensus mechanism (I defer on the real world likelihood of this result happening)

In case of consensus breaking, or if the optimistic phase is deemed unreliable, then the pessimistic phase can happen. There's some decisions how to implement this as well. The most straightforward is to give all seen ready messages to the consensus mechanism and then stop when complete (with maybe a timeout period to start over).

Alternatively if this is a lot of data then possibly the consensus mechanism can simply decide on which HybridVSS instances complete and then rely on nodes to gossip the actual HybridVSS results from exactly those instances to produce the final key. Or, maybe another alternative is to just use the first generated key signed by t+1 weight of nodes that appears in the consensus mechanism. In the case where nodes disagree on which HybridVSS instances completed, there doesn't appear to be any obvious issue with nodes signing all generated keys that appear valid to them (i.e. if a node sees >= t+1 weight instances of HybridVSS complete, it signs the resulting key. if a node sees another node sign a key and knows >=t+1 HybridVSS instances that result in that key, it signs that key too)

There will probably need to be careful proofs of the new HybridDKG protocol, but at a high level we should figure out the informal tradeoffs first.

We have to address the handoff between epochs. This is something which we have ignored up until now, but is the primary research question remaining: when the previous epoch ends, two things happen:

1. The previous DKG's public key instantly becomes invalid, therefore invalidating every client's state (until it gets the new public key) and invalidating all transactions in the mempool
2. The new DKG needs to run, which takes some time and therefore the network pauses processing transactions while the DKG is running. We could run the DKG before the new epoch begins, but then this creates new questions because the new DKG validator set and the current validator set may be different

The more performant the DKG, the less of an issue (a) and (b) are, but even in the best case it seems unavoidable to pause the network for 3-5 blocks, unless there is some clever pipelining or handoff trick we can use.

I believe we can and should run the DKG entirely during the previous epoch. This requires the staking phase to end slightly before the new epoch begins, and risks that the previous validator set can censor the DKG from proceeding (liveness risk) but I think this risk is very low when you consider:

1. the minimal potential gain from delaying the DKG
2. the large overlap in validator set between epochs; and
3. in the worst case, the DKG can begin as soon as the new epoch starts, causing at most several blocks delay in the network

Currently the dealer in HybridVSS has to construct and commit to an m^2 size polynomial (m is number of shares) which is fairly expensive to do. Unfortunately it seems nontrivial to improve this to subquadratic time, however since it is part of the DKG it might be an acceptable cost.

More importantly if we do a separate HybridVSS for every share, that is a lot of overhead, so we really only want to do HybridVSS once per identity (say, n is the number of nodes/identities) and nm^2 is a lot more manageable. This seems like a realistic proposal, but the details need to be carefully sketched out.

The first observation is that we can have HybridVSS issue multiple shares to each identity (likely no problem with this). The second observation is that HybridDKG now needs t+1 weighted HybridVSS instances to finish and not just t+1 identities to finish (the DKG proofs need to be modified accordingly)
The third observation is that some aggregation of messages may be possible (e.g. one ready message per HybridVSS per identity rather than per share) to potentially save more communication complexity and/or storage costs

The amount of code changes is relatively small but we mostly have to make sure we don't break any of the important properties of HybridDKG doing this.

Sikka has an initial implementation of threshold decryption at https://github.com/sikkatech/arkworks-threshold-decryption, that can be integrated easily.

This is a two-part task:

1. Implement key refresh with the same node identities and weights, this is done by running the same DKG again with final key 0. Should be straightforward - mostly just adding the functionality.
2. Design and implement key refresh with different weights (some nodes may have weight increase, others decrease). This is probably more challenging but worth investigation.

Currently, the benchmarks are rather ad-hoc. It would be nice to use criterion to scientifically microbenchmark each part of the protocol, so that we can experiment with changes and see if they're beneficial.

It's possible for the dealer to precompute, offline, many of the computationally expensive parts of its VSS. This feature can be added if it's judged to be useful.

A security analysis should be done to assess the risk of having VSS dealer/private data stored on the dealer's server for potentially a nontrivial amount of time.

Decrypting involves a number of pairing checks, as @simonmasson observes we can optimize that to two miller loops and one final exp (example.

Additionally since many transactions are getting decrypted at the same time, it would be nice to batch all these checks together (e.g. because you're verifying a bunch of decryptions made with the same key share).

Since tx decryption is going to need to be very fast it's worth looking at these optimizations

The crypto protocol draft should be reviewed for errors, details added as necessary, TODOs resolved, etc. Ideally the crypto side of the protocol should be well documented and clearly comparable to both the written code and also to the original papers.

The cryptoeconomics discussion may be added to the same document or a separate document later.

This is not the highest priority task, but it is relatively straightforward and can also be good for testing the overall system.

https://www.cc.gatech.edu/~aboldyre/papers/bold.pdf

Extremely high performance is a design goal of Ferveo, this will require some creativity to achieve. This issue is to track the discussion and ideas.

Relevant readings:
https://twitter.com/aniketpkate/status/1319345423811809291
https://people.csail.mit.edu/devadas/pubs/scalable_thresh.pdf

Tracking all major tasks and decisions to be made, both roughly in order of importance.

Decisions:

• Choose a BLS signature crate/library/implementation and/or choose to write a new one #23
• Choose symmetric key crypto primitives #10
• Choose signature scheme for DKG participant identities #11

Tasks:

• Implement integrate BLS signatures #23
• Implement group secret key compatible signature (if necessary)
• Integrate threshold decryption crate and/or contribute upstream #26
• Side-channel analysis #16
• Implement loading of trusted setup #17
• Implement all scalability and performance enhancements #5
• Implement key refresh #18
• Implement randomness beacon #27
• #54
• #77
• #78
• #79

We're also dependent on some external tasks:

• Improve arkworks Fp12 performance arkworks-rs/algebra#121
• Improve arkworks subgroup check #14
• Contribute tests to arkworks hash-to-curve arkworks-rs/algebra#343
• Use hash-to-field/hash-to-curve to arkworks arkworks-rs/algebra#343

Unlike #10 there is really only one choice for public key primitives, BLS12-381 curve with the described key agreement/encryption/signature schemes. This is mostly dictated by the small number of acceptable choices and also desire to reuse existing trusted setups.

Nevertheless, alternatives should be discussed and choices finalized, and then implementations of primitives written and/or imported from existing crates.

Unless there are good crates that exist, implement the fancy KZG commitment schemes for high performance and small multi-polynomial multi-value opening proof sizes.
https://github.com/khovratovich/Kate/blob/master/Kate_amortized.pdf

If KZG commitments are used, then a trusted setup has to be loaded (probably an existing setup, Zcash or Filecoin)

This is a straightforward task, implement in Rust the download/verification/loading of the trusted setup. Ideally there should be extensive opportunity for code reuse/external crates here, since for example Zcash clients perform this task exactly.

Currently, there are quite a few expensive loops which should be parallelized with rayon.

I'm not sure if the following has been discussed or proposed, but I'm putting this into an issue here so we can have a place to discuss things related to this topic.

Once Ferveo is implemented and integrated into the consensus system. It should not be hard to also implement the following:

(1) Threshold (BLS?) signatures
(2) Compact certificates of finalized states using threshold signatures.
(3) Light clients that rely on verification of public key updates and state certificates.

Currently, it is assumed that every VSS transcript is added to a block, which incurs a lot of overhead in the case of many validators.

When the gossip mechanism is redesigned, Ferveo should do VSS aggegation (basically, the aggregatable DKG approach) within the gossip layer, incurring cost more like O(# key shares * log(# validators))

When the aggregate step of the PVSS fails, the node handling the aggregation has to identify which PVSS instance was bad, and then send a complaint message to everyone else.

The protocol extensively sends/receives points on the BLS12-381 curve from third parties, there may be many subgroup checks needed to ensure that all such points lie in the prime order subgroup.

This task involves two parts:

1. Identify every place in the protocol where such subgroup checks are necessary
2. Implement a fast subgroup check algorithm, e.g. https://eprint.iacr.org/2019/814.pdf

Hopefully we can identify a suitable existing implementation, otherwise implement independently.

Any implementation should take into account the issues described here

https://ethresear.ch/t/fast-verification-of-multiple-bls-signatures/5407

NuBLS is a pure rust implementation, but of course the dependencies are an issue.
blstrs - another option, dependency issues and also depends on unsafe code/C library

https://www.cc.gatech.edu/~aboldyre/papers/bold.pdf

Some parameters such as:

• $n$, total number of nodes/identities in the protocol
• $W$, total weight of all nodes
• $f$, maximum number of crashed nodes at any time
• $T$, maximum timeout before changing phases in the protocol

will need to be determined by experimentation once the protocol is implemented and initial performance tweaks are added. Experiments might also suggest changes to the abstract protocol in addition to the parameters.

While not an immediate issue, explore the possibility of using arkworks algebra for polynomial types and curves.

I have two primary reasons to consider this:

1. I'd rather not use the nalgebra crate...it's probably good at what it does, but it's rather difficult to use for our purposes, and arkworks has a nice univariate and multivariate polynomial type.
2. It would be nice to avoid dependency fragmentation. There are already some chess moves with dependencies right now (e.g. dusk bls12-381 for FFT, possible use of forks of ff with assembly language implementations, etc etc). A single stable upstream which we can easily submit PRs to would be nice.

We need tests for the protocol and implementation.

[ ] End-to-end tests which run the entire DKG and threshold encryption/decryption from beginning to end
[ ] Tests for each individual component, to get 100% coverage of code paths
[ ] Test vectors for components where appropriate

• Refactor signed messages into reusable Signed<T : Serializable> type
• Switch tests to use seeded/deterministic RNG
• NIZKP use hash to field