saitotech / saito-rust Goto Github PK
View Code? Open in Web Editor NEWA high-performance (reference) implementation of Saito in Rust
A high-performance (reference) implementation of Saito in Rust
We are using the serializers branch as a place to benchmark and look at code for serialization.
We are using bincode for now because it's easist, but the thinking is that we need to either use protobufs or do custom serialization.
I think the benchmarks for this should be done in a decent way so that we can actually use them as benchmarks and devtools going forward. It would be nice to perhaps also add benchmarking to a CI pipeline in the future.
Acceptance Criteria:
Subtasks
update_genesis_period()
& fork_id
to blockchain on_chain_reorganization()
itself.add_block_success()
update.Hey folks,
There has been some internal discussion about serialization and encoding, so I thought it would make sense to document some of the ideas in these discussions more publicly.
Rust gives quite a few methods to implement serialization, and currently there's a short list of the ones that have been discussed, each containing their own sets of pros and cons.
google/tarpc
a Rust implementation of RPC that requires no additional tooling or languages to create a message schema.Serde
using the bincode
crate.There's a few other open-ended questions to factor into this conversation that will influence the decision. One of them in my mind is will the Rust version be the only version that is allowed to run consensus? If so, being as tightly coupled to Rust isn't as big of an issue.
Personally, at time of writing I'm leaning most towards Protocol buffers. It's a good compromise of dev complexity, language agnosticism, and efficiency that gives us decent flexibility moving forward.
If there are other standards that should be considered, please mention them! Now is the time to lay it out before anything gets implemented.
Routing Hops are currently 130 bytes on average. They each contain two publickeys (from, to) and a signature. The signature is generated using the to address and a piece of data associated with the transaction.
Optimizing the size of routing paths is important over the long-term given the number of transactions that will be included in blocks. One possible modification is removing the "from" publickey because it can be sourced from the "to" field of the previous routing hop. The first routing "from" address can be found as the sender of the first INPUT slip who would have needed to sign the transaction.
We can explore using a commutive encryption algorithm as well -- one where only a single signature is needed and that signature is generated and swapped-out as additional paths.
Possible savings are 33 bytes per routing hop, possibly the final "to" publickey as well (as the block creator will contain the publickey). And all but one signature, saving 64 bytes per block. Average data sizes per hop can be reduced considerably. This is well worth looking into once the software is up and operating reliably.
Currently running saito requires that there be a configuration file present under the current directory/configuration however this may not match realistic expectations when running saito in production. When executing the saito binary from a different directory it panics with thread 'main' panicked at 'Failed to read configuration.: configuration file "/Users/zyler/github/saito-live/configuration/base" not found', /Users/zyler/github/saito-live/saito-rust/src/consensus.rs:136:44
Options for handling this gracefully would be:
Sane defaults without a configuration file
Allow path to configuration file to be specified via command line ie saito --config=/path/to/config.yml
or via environment variable ie $SAITO_CONFIG_PATH
If file not found search for configuration in a user-standard directory such as $HOME/.saito
If user config directory is not found search in os-specific config path for instance /etc/saito on linux
This error I encountered when using homebrew to install saito on my mac, and realized the node operator onboarding experience could be improved with better configuration options.
(https://github.com/saito-live/homebrew-saito to try installation via homebrew on mac.)
There's been some discussion about how we name things in the codebase, so I wanted to open up a discussion so that we could stick to some common conventions.
By default, we match the Rust styling: https://doc.rust-lang.org/1.0.0/style/README.html
In general, we should fully type all names unless the name runs significantly long. We don't cut corners by shortening variables or attributes, so block
over blk
and transaction
over tx
. In the future, well defined acronyms can describe processes like automatice transaction rebroadcasting
(ATR
for short).
Getter functions just reference the piece of data that we want access to -> id() returns id. This also applies to a function like latest_block
in Blockchain
.
Setter functions use the convention of set_${attribute}
.
Additional functions that perform a more complex transform, or some other complex functionality will start with a verb, then describe what the function does. If there's not a guarantee that the operation will be successful, we use the verb try
. Only if we're sure of the success of the operation would this be negated.
Here's an example with Blockchain
. When we want to add a block to the blockchain, the function we call is try_add_block
, and not add_block
. The function add_block
is only called when we are certain that the block should be added to the blockchain. In a similar vein, when we attempt to bundle a block in Mempool
we call try_bundle_block
, and only call bundle_block
when the proper requirements are met.
For our primitive structs, we want to distinguish between Primitive
and PrimitiveCore
. Core
version of the struct will be what's sent over the wire between nodes, and represents the simplest version of the struct. The full-bodied version is available to allow for each node to enrich its data.
We could use a real logging library, it would be nice to be able to have log levels and be able to pipe log message easily.
I run the node I get listening on http://127.0.0.1:3030
how to connect to other nodes?
Running cargo test
on stable results in many tests not finishing within 60 seconds, and seem to go for many minutes.
...
test network::tests::test_blockchain_causes_sndblkhd has been running for over 60 seconds
test network::tests::test_network_message_sending has been running for over 60 seconds
test network::tests::test_sndblkhd has been running for over 60 seconds
test network::tests::test_sndchain has been running for over 60 seconds
test networking::api_message::tests::test_message_serialize has been running for over 60 seconds
test networking::message_types::handshake_challenge::tests::test_challenge_serialize has been running for over 60 seconds
test networking::message_types::request_block_message::tests::test_request_block_message_serialize has been running for over 60 seconds
test networking::message_types::request_blockchain_message::tests::test_request_blockchain_message_serialize has been running for over 60 seconds
test networking::message_types::send_block_head_message::tests::test_send_block_head_message_serialize has been running for over 60 seconds
test networking::message_types::send_blockchain_message::tests::test_send_blockchain_message_serialize has been running for over 60 seconds
test slip::tests::slip_addition_and_removal_from_utxoset has been running for over 60 seconds
test staking::tests::blockchain_roll_forward_staking_table_test_with_test_manager has been running for over 60 seconds
test staking::tests::blockchain_staking_deposits_test has been running for over 60 seconds
test staking::tests::staking_create_blockchain_with_many_staking_deposits_many_staker_payouts_per_block has been running for over 60 seconds
test staking::tests::staking_create_blockchain_with_two_staking_deposits_one_staker_payout_per_block has been running for over 60 seconds
test storage::tests::write_read_block_to_file_test has been running for over 60 seconds
Once there are golden tickets #36 being produced in consensus, the system will start to have slips that will govern network functions.
Each node needs a way to collect and organize their slips, as well as create transactions with the proper slip balances to pay fees to the network. We'll want a Wallet
struct that will be capable of doing this functionality.
The Wallet
will need to contain:
Arc
and Rwlock
)The basic API we should support at first is
add_slip(slip: Slip)
remove_slip(slip: Slip)
create_transaction(fee: u64, amount: u64)
create_signature(data: [u8; 32])
A rough implementation can be found in the master
branch of this repo.
I want to create some tooling so that we can actually simulate creating a blockchain and then hitting it with incoming blocks and transactions and observe the behavior. If Cargo bench can do all this, I think we shoudl use that. Otherwise, maybe this should a separate crate under a tools lib that interacts with the node runtime via the system(e.g. through some socket)
After @clayrab pointed out in #42, we'll want a generic way to create randomness in golden ticket creation. Traditionally, we've used a SHA256 miner to achieve this, but we'll want some sort of interface for multiple lottery games to conform to outside of just mining.
The way to go about this is creating a trait called LotteryGame
that exposes an interface that all Saito lottery games need to conform to. A higher level struct called Lottery
will control running the game, and will be used to initialize the LotteryGame
when a node starts running.
For now, to generate slips and get Saito circulating in the system, we auto generate GoldenTicket
Transaction
s and feed them into the Mempool
where they're bundled and added to the chain.
We want a Miner
module to do some mining work to generate these golden tickets. We want to port over the existing Miner
from the master
branch, remove the Lottery
and LotteryGame
cruft around it and add tests.
https://github.com/SaitoTech/saito-rust/runs/3468793166#step:3:1076
error: this boolean expression contains a logic bug
--> src/blockchain.rs:315:16
|
315 | if save_to_disk || true {
| ^^^^^^^^^^^^^^^^^^^^ help: it would look like the following: `true`
|
= note: `#[deny(clippy::logic_bug)]` on by default
help: this expression can be optimized out by applying boolean operations to the outer expression
--> src/blockchain.rs:315:16
|
315 | if save_to_disk || true {
| ^^^^^^^^^^^^
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#logic_bug
To extend adding blocks to the longest chain, we'll need to implement saving Block
s to disk. We'll want a Storage
module to easily read and write Block
s.
As we're using tokio
, the best way to do this would be to implement this asynchronously. We'll also want to extend some traits within Block
to make it easy to swap out serialization, as we've yet to decide what serialization method we'll be implementing in the long term. For now, using bincode to serialize should be fine, as long as it's implemented in a trait and not hardcoded.
The functions we want to add are:
async read_block_from_disk(block_hash: [u8: 32]) -> Result<Block, Err>
async write_block_to_disk(block: Block) -> Result<(), Err>
A synchronous, rough implementation of Storage
can be found in the master
branch.
Acceptance Criteria:
Subtasks:
I think the implementation of add_block_to_blockchain will much simpler if we get the datastructures right.
I believe a having a map of block_id -> block_hash which represents the longest chain will make things must easier.
This should be in it's own class with only a handful of methods:
rollforward(newblock)
rollbackward() -> hashOfRemovedBlock
getBlockById(u64) -> hashOfSomeBlockInTheLC
getLatestBlock() -> hashOfLatestBlock
bottomOfEpochBlock() -> hashOfBlockThatsAboutToFallOffTheChain
At the top of the longest chain, we would need stack-like behavior. I.E. the latest block is pushed and popped in a LIFO manner.
At the bottom of the epoch, we'll also need to push and pop as blocks fall off the epoch in a FIFO(queue-like) manner and also keep a cache of recently-fallen-off blocks which can then be pushed back into the bottom of the epoch when we do rollback().
The simplest way I can think to handle all this is to have two stacks, one which has the latest block at the top, and another which holds the 2nd-oldest epoch. Since the length of the epoch is fixed, it's straightforward to use something like a ring-buffer in a fixed-length array to acheive this. When we rollforward, the block which is 2 epochs down the longest chain is simply lost. This supports a rollback the length of the entire epoch. We can worry about optimizing this later.
Hi team, I've just noticed that we're using the outdated jemallocator.
Check this PR from rust-core for more details.
Updated jemalloc lib
This updated one is from tikv
TL;DR: this upgrade could help for sized deallocation's usage correctly, particularly to rustdoc.
So somehow it can deliver a sizable performance increase. We can do some benchmarks after that.
Lets consider my suggestion. Thank you.
PEERS_DB_GLOBAL
& its usagePEERS_REQUEST_RESPONSES_GLOBAL
& its usagePEERS_REQUEST_WAKERS_GLOBAL
& its usageINBOUND_PEER_CONNECTIONS_GLOBAL
& its usageOUTBOUND_PEER_CONNECTIONS_GLOBAL
& its usageWe have a keypair class that enables us to create keypairs, but the UX needs to be done.
Our vision is that users should be prompted to create a password encrypted keypair the first time they start a node which creates an encryted keypair file at a standard location.
If the keypair file already exists, the user should be prompted for the password.
The user should also be able to pass a command line argument with the path to another encryted keypair file(which should use the same password encrypting scheme and prompt them for a password also).
Another option might be to allow users to pass an unencrypted password file as a command line parameter, but this is I think just "nice to have". Could be done if it's very easy, but might also complicate UX significantly.
We're open to suggestions on changes to this design.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.