GithubHelp home page GithubHelp logo

zcashfoundation / zebra Goto Github PK

View Code? Open in Web Editor NEW
401.0 22.0 92.0 28.39 MB

Zcash - Financial Privacy in Rust ๐Ÿฆ“

Home Page: https://zfnd.org/zebra/

License: Apache License 2.0

Rust 99.29% HTML 0.01% Shell 0.57% Dockerfile 0.12% JavaScript 0.01% CSS 0.01%
zcash rust zebra zcash-node

zebra's Introduction

Zebra logotype


CI Docker CI OSes Continuous Delivery codecov Build docs License

Contents

About

Zebra is the Zcash Foundation's independent, consensus-compatible implementation of a Zcash node.

Zebra's network stack is interoperable with zcashd, and Zebra implements all the features required to reach Zcash network consensus, including the validation of all the consensus rules for the NU5 network upgrade. Here are some benefits of Zebra.

Zebra validates blocks and transactions, but needs extra software to generate them:

  • To generate transactions, run Zebra with lightwalletd.
  • To generate blocks, use a mining pool or miner with Zebra's mining JSON-RPCs. Currently Zebra can only send mining rewards to a single fixed address. To distribute rewards, use mining software that creates its own distribution transactions, a light wallet or the zcashd wallet.

Please join us on Discord if you'd like to find out more or get involved!

Getting Started

You can run Zebra using our Docker image or you can build it manually. Please see the System Requirements section in the Zebra book for system requirements.

Docker

This command will run our latest release, and sync it to the tip:

docker run zfnd/zebra:latest

For more information, read our Docker documentation.

Building Zebra

Building Zebra requires Rust, libclang, and a C++ compiler.

Zebra is tested with the latest stable Rust version. Earlier versions are not supported or tested. Any Zebra release can start depending on new features in the latest stable Rust.

Around every 6 weeks, we release a new Zebra version.

Below are quick summaries for installing the dependencies on your machine.

General instructions for installing dependencies

  1. Install cargo and rustc.

  2. Install Zebra's build dependencies:

    • libclang is a library that might have different names depending on your package manager. Typical names are libclang, libclang-dev, llvm, or llvm-dev.
    • clang or another C++ compiler: g++ (all platforms) or Xcode (macOS).
    • protoc

[!NOTE] Zebra uses the --experimental_allow_proto3_optional flag with protoc during compilation. This flag was introduced in Protocol Buffers v3.12.0 released in May 16, 2020, so make sure you're not using a version of protoc older than 3.12.

Dependencies on Arch

sudo pacman -S rust clang protobuf

Note that the package clang includes libclang as well as the C++ compiler.

Once the dependencies are in place, you can build and install Zebra:

cargo install --locked zebrad

You can start Zebra by

zebrad start

See the Installing Zebra and Running Zebra sections in the book for more details.

Optional Configs & Features

Initializing Configuration File
zebrad generate -o ~/.config/zebrad.toml

The above command places the generated zebrad.toml config file in the default preferences directory of Linux. For other OSes default locations see here.

Configuring Progress Bars

Configure tracing.progress_bar in your zebrad.toml to show key metrics in the terminal using progress bars. When progress bars are active, Zebra automatically sends logs to a file.

There is a known issue where progress bar estimates become extremely large.

In future releases, the progress_bar = "summary" config will show a few key metrics, and the "detailed" config will show all available metrics. Please let us know which metrics are important to you!

Configuring Mining

Zebra can be configured for mining by passing a MINER_ADDRESS and port mapping to Docker. See the mining support docs for more details.

Custom Build Features

You can also build Zebra with additional Cargo features:

You can combine multiple features by listing them as parameters of the --features flag:

cargo install --features="<feature1> <feature2> ..." ...

Our full list of experimental and developer features is in the API documentation.

Some debugging and monitoring features are disabled in release builds to increase performance.

Known Issues

There are a few bugs in Zebra that we're still working on fixing:

Documentation

The Zcash Foundation maintains the following resources documenting Zebra:

User support

For bug reports please open a bug report ticket in the Zebra repository.

Alternatively by chat, Join the Zcash Foundation Discord Server and find the #zebra-support channel.

Security

Zebra has a responsible disclosure policy, which we encourage security researchers to follow.

License

Zebra is distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE and LICENSE-MIT.

Some Zebra crates are distributed under the MIT license only, because some of their code was originally from MIT-licensed projects. See each crate's directory for details.

zebra's People

Contributors

aarnott avatar arya2 avatar bishopcheckmate avatar chairulakmal avatar conradoplg avatar dconnolly avatar dependabot-preview[bot] avatar dependabot[bot] avatar ebfull avatar elijahhampton avatar fanatid avatar gtank avatar gustavovalverde avatar hdevalence avatar idky137 avatar jackgavigan avatar jvff avatar kiminuo avatar mpguerra avatar oxarbitrage avatar preston-evans98 avatar rex4539 avatar sandakersmann avatar str4d avatar teor2345 avatar tony-iqlusion avatar upbqdn avatar vramana avatar yaahc avatar zancas avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zebra's Issues

Outgoing `Ping` messages

In #17 we handle incoming ping messages at the edge, but we also need to generate outgoing ping messages in case the remote peer does not send them to us. This involves using a timer to generate a stream of events (presumably an Interval), sending a ping message on every time interval event, and checking that the server receives a pong before some timeout.

Currently the PeerServer state machine is already somewhat complicated, but I think a relatively clean way to do this functionality would be to take advantage of the fact that the PeerClient uses an mpsc channel to communicate with the PeerServer, so there is no limitation that the PeerServer must have only one client. So when we spawn the server task, we could also spawn a second client task that waits on a timer interval and sends a new Request::Ping(Nonce) to the server task.

The downside of this approach is that we would have to add a Ping variant to the Request enum, which feels like a slight layering violation (these are really messages about transport maintenance, not messages sent over that transport) but I think it would otherwise integrate cleanly with the code we have.

Implement a `PeerSet` load-balancer and outgoing connection router.

Using the Discover implementation of #47, the PeerSet should be able to discover new (load-instrumented) peers, and provide a Service implementation usable by the rest of the application to make outgoing requests. Those requests should be load-balanced over the available (ready) peers, probably using power-of-two-choices as in tower-balance (which we cannot use directly because we have bidirectional loading from the p2p setting, not unidirectional loading as in the conventional tower service setting).

Encapsulate all peer-related objects into a network abstraction

We would like to encapsulate all peer-related objects into a single abstraction for "the rest of the network", which we can expose from zebra-network while keeping the PeerClient, PeerServer, etc. as private implementation details. This provides a cleaner API (and allows us to finish #37).

Currently the PeerSet already mostly represents this, but it only encapsulates some functionality, as constructing a PeerSet requires:

  1. constructing a TimestampCollector;
  2. constructing an internal service of type S: Clone + Service<Request> to handle peer requests;
  3. constructing a PeerConnector (using the timestamp collector);
  4. constructing an appropriate Discover instance (just meaning one that satisfies the trait bounds, not all of #47 );
  5. constructing the PeerSet itself;
  6. per #47, making that Discover instance handle incoming connections (unfinished, needs #46, progress in #68);
  7. per #47, making that Discover instance able to supply new peers (unfinished, needs #45, #49);

Of this, (2) seems irreducible: the internal service should not be constructed by the PeerSet, it should be supplied, because it represents "this node" in the "this node / the network" distinction, but all of the others seem like components that could be initialized by the PeerSet itself, so that the process would look like:

  1. Construct an internal service of type S: Clone + Service<Request> to handle peer requests;
  2. Pass S and a config to a PeerSetBuilder and finalize to obtain (potentially a wrapper of) a PeerSet;
  3. Use the resulting object as a Service<Request> that sends requests to "the network".

Getting to this point requires finishing off the functionality referenced above, as well as encapsulating all of the component setup, but parts 1-5 are already contained in the connect stub and could be refactored into a PeerSet builder in parallel to completion of the rest of the functionality.

Once this is done, #54 would consist only of defining an internal service that responds to GetPeers requests using the PeerSet's TimestampCollector and passing it to the PeerSet constructor.

Add an Amount type.

This should be a newtype wrapper around i64 that enforces range constraints. This means that arithmetic operations are fallible, so working with Amounts will require lifting into the Option monad, but by careful implementation of the ops traits, it should be possible to make this relatively ergonomic.

Add a networking config struct.

Currently we handle configuration data using Abscissa's config handling functionality, which works by defining config structs that are serde-compatible, so that loading a config is done by, e.g., deserializing from a toml file.

The networking code contains various parameters that should be configurable, and which need to be configured from inside zebra-network (which does not depend on abscissa) to obey encapsulation. However, these options should ideally be able to be loaded from zebra (which does use abscissa) in the normal way.

I think that if we define a Config struct with the required fields in zebra-network and derive the serde traits for it, we should be able to add it as a section in zebrad::ZebradConfig and have it work correctly.

Finish refining all primitive types brought in by `Transaction` data.

Following from #105 (comment):

  1. Many of the types in the structures that make up a Transaction need to be refined to assign them semantic meaning. The complete list can be found by running rg 'XXX refine' -A 2 in zebra/zebra-chain/src. Fleshing out many of these types will require further work โ€“ for instance, the Zcash-flavored Ed25519 pubkey in the JoinSplit data requires #109.

This issue should be closed when there are no more refinements left to do. Once #105 is merged, I will replace this description with a checklist of types to refine.

  • an Amount type (#126)
zebra-chain/src/transaction.rs
81:        // XXX refine this to an Amount type.
82-        value_balance: i64,
zebra-chain/src/transaction/joinsplit.rs
54:    /// XXX refine to an Amount
55-    vpub_old: u64,
--
59:    /// XXX refine to an Amount
60-    vpub_new: u64,
zebra-chain/src/transaction/transparent.rs
51:    // XXX refine to Amount ?
52-    pub value: u64,
53-
  • a (Sprout?) (Sapling?) nullifier type (#287, #288)
zebra-chain/src/transaction/joinsplit.rs
16:    /// XXX refine type
17-    pub nullifier: [u8; 32],
zebra-chain/src/transaction/shielded_data.rs
17:    /// XXX refine to a specific type.
18-    pub nullifier: [u8; 32],

One of these is a nullifier for Sprout, while the other is for Sapling โ€“ should these be the same type or different types?

  • a MAC type
zebra-chain/src/transaction/joinsplit.rs
20:    /// XXX refine type
21-    pub vmac: [u8; 32],
  • note commitment type (sprout) (#36)
zebra-chain/src/transaction/joinsplit.rs
37:    /// XXX refine type
38-    pub commitment: [u8; 32],
  • types related to note encryption (sprout):
zebra-chain/src/transaction/joinsplit.rs
41:    /// XXX refine type
42-    /// XXX this should be a [u8; 601] but we need trait impls.
43-    pub enc_ciphertext: Vec<u8>,
  • a sprout commitment tree root (#36)
zebra-chain/src/transaction/joinsplit.rs
65:    /// XXX refine type
66-    anchor: [u8; 32],
  • an X25519 key (#313):
zebra-chain/src/transaction/joinsplit.rs
69:    /// XXX refine to an x25519-dalek type?
70-    ephemeral_key: [u8; 32],
  • an Ed25519 key (#109)
zebra-chain/src/transaction/joinsplit.rs
88:    // XXX refine to a Zcash-flavored Ed25519 pubkey.
89-    pub pub_key: [u8; 32],
90-    /// The JoinSplit signature.
91:    // XXX refine to a Zcash-flavored Ed25519 signature.
92-    // for now it's [u64; 8] rather than [u8; 64] to get trait impls
93-    pub sig: [u64; 8],
  • a sapling value commitment type:
zebra-chain/src/transaction/shielded_data.rs
11:    /// XXX refine to a specific type.
12-    pub cv: [u8; 32],
  • RedJubjub types (#125)
--
21:    /// XXX refine to a specific type.
22-    pub rk: [u8; 32],
23-    /// The ZK spend proof.
--
30:    /// XXX refine to a specific type: redjubjub signature?
31-    /// XXX for now it's [u64; 8] instead of [u8; 64] to get trait impls
32-    pub spend_auth_sig: [u64; 8],
--
77:    // XXX refine this type to a RedJubjub signature.
78-    // for now it's [u64; 8] rather than [u8; 64] to get trait impls
79-    pub binding_sig: [u64; 8],
  • Jubjub key agreement types: #361
zebra-chain/src/transaction/shielded_data.rs
50:    /// XXX refine to a specific type.
51-    pub ephemeral_key: [u8; 32],
  • note commitment type (sapling) (#36)
zebra-chain/src/transaction/shielded_data.rs
46:    /// XXX refine to a specific type.
47-    pub cmu: [u8; 32],
  • note encryption (sapling):
zebra-chain/src/transaction/shielded_data.rs
54:    /// XXX refine to a specific type.
55-    /// XXX this is a Vec<u8> rather than a [u8; 580] to get trait impls
56-    pub enc_ciphertext: Vec<u8>,
--
59:    /// XXX refine to a specific type.
60-    /// XXX this is a [u64; 10] rather than a [u8; 80] to get trait impls
61-    pub out_ciphertext: [u64; 10],
--

Extract and refactor handshake logic from `connect` stub into some kind of `Peer`.

Currently the connect stub has basic handshake logic. We should extract, refactor, and encapsulate this logic into some kind of a Peer, responsible for handling the stateful handling of Bitcoin messages.

It's kind of an open question how these Peers should communicate with the rest of the node; one idea that seems promising is to construct an internal request/response protocol and make the Peers responsible for translating the legacy network protocol into requests in the internal protocol, passing those to some tower-based services internally, and then mapping the responses back into Bitcoin messages. But this is only a general concept, and doesn't deal with the actual connection mechanics (async channels? references to tower services?).

Implement transaction serialization.

Following from #105 (comment):

  1. The actual transaction encoding and decoding is not implemented, although there is a stub module in which it will live (transaction/serialization.rs). Implementing ZcashSerialize and ZcashDeserialize is a further chunk of work, but because all of the unrefined types are in place, it can be done independently of (1) by, e.g., parsing something as a [u8; 32] instead of some refined PublicKey type.

This issue is for implementing serialization for transactions. It probably makes sense to do this together with the next follow-on item from that PR:

  1. The as-yet-unimplemented encoding and decoding is not tested. In addition to the transaction test vectors, we could define a proptest strategy to generate random transaction data to check that the encodings round-trip.

Analyze overlap between `zebra_chain` and `zcash_primitives`

Now that the zcash_primitives crate has been published and we have mostly finished the networking code, we should analyze the overlap between zcash_primitives and zebra_chain, and decide:

  1. What (intended) functionality is common between these two crates;
  2. Whether we should re-export parts of zcash_primitives into zebra_chain;
  3. Whether we should combine parts of zebra_chain into zcash_primitives and upstream the changes.

Add a peer listener.

In addition to dialing outgoing peers, we need to spawn a TCP listener task that listens for incoming connections and completes a handshake with them, producing a new PeerClient.

Update to Tokio 0.2

As of now, hit Poll::Pending because the most recent hyper, 0.13.0-alpha.4, requires = 0.2.0-alpha.6.

Add an address book to track peer information.

To respond to getaddr messages we need to maintain an address book with liveness information about network peers, to be introduced in #23 .

This address book could also be responsible for maintaining ownership of peers or routing requests to and from peers, but it's not clear that that would be the best design (compared to, e.g., handing ownership of peers to the tokio runtime and having it drive them to completion, or using tower to do load-balancing over peers, etc.), so it seems better to leave those tasks for now and have an address book solely responsible for tracking addresses and liveness information.

One idea of how to do this is to create an async mpsc channel in the addressbook and store a transmission handle to it with each peer, then use StreamExt::then to hook a timestamp message transmission step into the peer's message stream processing.

Zcash-flavored Ed25519

Parts of the Zcash specification require Ed25519 with specific, consensus-critical verification rules, for instance in the JoinSplit signatures. Because it is critical that we do Ed25519 verification exactly as in the spec, we may want to create a Zcash-flavored Ed25519 on top of curve25519-dalek that has exactly the properties we need.

Proptest scaffold

Before doing #27 it would be convenient just to have a basic proptest skeleton so that the work of #27 or other property tests is just the work of writing the tests themselves.

Clean module layouts and imports.

As we flesh out the code, we're placing things in various parts of the module tree and importing various items from various paths, but as things take a clearer form we will have a better idea of the ideal logical structure of the module tree. Before we do any release, we should do a reorganization pass to ensure that the module tree is conceptually coherent, and probably use re-exports to simplify presentation of the external module components to external crates. However, it may not make sense to do this right now, while things are still in flux.

Handle `PeerClient` timeouts correctly.

The current peer handling code from #17 does not handle client request timeouts correctly (or at all). This must be fixed. Because we need to statefully translate the legacy protocol into our internal protocol, I think we cannot use the tower_timeout middleware, because we need to reset the PeerServer state when the request is dropped.

There are three ways we could proceed:

  1. "Drop": after a fixed timeout period, if a response has not arrived, fail the peer server and client together, and possibly spin up a new connection to the same remote in a later round of service discovery. This avoids having to change the server's state machine.

  2. "Server Timeout": add a timeout event to the server's state machine that fails a client request after a particular time interval (presumably a configured value set by the PeerConnector).

  3. "Something with Tower": come up with a way to make the server event loop work gracefully with tower_timeout middleware (e.g., detect that the request has been dropped? unsure how this would trigger an event... or if this option is possible).

Create `NextPeerService`

A service or component that will manage metadata about the available peers and their health/availability/latency, and a method or main service call that will return the best/next peer to connect to.

Design Sprout note commitment trees.

We may need a sprout note commitment tree, even if we checkpoint on sapling activation, in order to do sprout-on-groth16 proofs to move old sprout value.

Figure out whether `zebra-reactor` still makes sense with Abscissa

Originally zebra-network was going to have all of the networking abstractions, while zebra-reactor would have all of the components that drive them (with respect to the state in the storage layer, etc). However, it may make sense to fold those concepts into zebrad directly, because Abscissa already has built-in mechanisms for having a component registry, dependency injection, global access to config state, etc.

Transaction naming scheme

Many parts of the spec use tx to refer to a transaction, either standalone or in compound names (e.g., tx_in, tx_out, and more as in #13).

I think we should decide whether or not to adopt this convention, but whichever way we go, I think we should be consistent, so that if we use tx in some places we should use it everywhere (e.g., Tx rather than Transaction, TxHash rather than TransactionHash, etc), or else not use it and always use "transaction".

Personally I would prefer to use tx but I would more strongly prefer consistency.

Use property testing for message struct serialization round-trips

We should try to use property-based testing (probably proptest) for checking that message (de)serialization round-trips. This has structured input, so in contrast to fuzzing (unstructured input), this is probably less useful for checking that we cannot be vulnerable to malformed messages, but it's probably more useful for checking that the encoding functions actually work.

RedJubJub signatures

In addition to #109, we also need a library for RedJubJub signatures. One exists in librustzcash/zcash_primitives but it is coupled to the rest of that library, and the zkcrypto/jubjub implementation will soon be using bls12_381 directly.

Revisit `Debug` implementations

We should do a pass over the existing Debug implementations so they appear nicer in tracing output.

For instance, instead of

BlockHeaderHash([247, 3, 134, 210, 114, 134, 208, 233, 93, 156, 161, 56, 104, 238, 53, 170, 136, 70, 17, 174, 178, 248, 161, 167, 29, 125, 152, 0, 0, 0, 0, 0])

we could have a custom Debug impl that uses hex to do, e.g.,

BlockHeaderHash(81cd02ab7e569e8bcd9317e2fe99f2de44d49ab2b8851ba4a308000000000000)

This also allows using tracing filters on hash values, etc.

Add an Abscissa command for DNS seeder information.

Our MVP to test our new networking design is a daemon that can power a DNS seeder; this is a trivial application but it is measurable and concrete and tests that the networking design does actually work in a visible way, before we complete the rest of the work of the node implementation.

Currently the only command we have is the connect command, which exists solely for debugging during development and is intended to be thrown away.

To run the daemon we need to have a different command. This could either be called run (the future command to start a full node should certainly be run), or something else to indicate that the command only does address spidering and that the rest of the node functionality is not yet implemented. Thoughts?

Add a helper method for reading fixed-size byte arrays

From adc421f

The deserialization logic for message headers is somewhat ugly because of a lack of a convenient idiom for reading a few bytes at a time into a fixed-size array. Perhaps this could be fixed with an extension trait, or avoided altogether by using u32s instead of [u8; 4], even when the latter is the "morally correct type".

Making fixed size arrays of generic length is awkward (until const generics arrive... someday), but I think we could add read_4bytes, read_12bytes etc methods to the ReadZcashExt extension trait for each size we need -- I don't think we'll ever need more than a few sizes.

Formerly part of #8.

Checksum(Read|Writ)er

Re: adc421f:

Deserialization does an extra allocation, copying the body into a temporary buffer. This avoids two problems: 1) capping the number of bytes that can be read by the Reader passed into the body parser and 2) allows making two passes over the body data, one to parse it and one to compute the checksum.

We could avoid making two passes over the body by computing the checksum simultaneously with the parsing. A convenient way to do this would be to define a

struct ChecksumReader<R: Read> {
    inner: R,
    digest: Sha256,
}

impl<R: Read> Read for ChecksumReader<R> { /* ... */ }

and implement Read for ChecksumReader to forward reads from the inner reader, copying data into the digest object as it does so. (It could also have a maximum length to enforce that we do not read past the nominal end of the body).

A similar ChecksumWriter could be used during serialization, although because the checksum is at the beginning rather than the end of the message it does not help avoid an allocation there. It could also be generic over a Digest implementation, although because we need a truncated double-SHA256 we would have to write a custom Digest implementation, so this is probably not worthwhile unless we have other checksum types.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.