Sharding manager contract, and related software and tests
Refer Py-EVM for the latest implementation progress.
See the "docs" directory for documentation and EIPs.
Please visit ethresear.ch.
This has a central role in the document, but it is an abstract word. Please add a short intro explaining what a Collation is.
In the Sharding Specification in paragraph of Line 276 it states "At the end, there is an additional step, finalizing the collation (to give the collator the reward...", then then explains the additional step. It says that after the network replies it will "apply the reward" to the collator, and then says that "The collator can then package up the collation, ...".
Why are we applying the reward before instead of after the collator has finished packaging up the collation?
https://ethresear.ch/t/the-stateless-client-concept/172
state_root of the pivot point collation from validator manager contractSome tests like tests/contract/test_log_emission.py, test_submit_vote.py, fast_forward is still pretty slow.
Not sure yet.
It's better to have type annotations here.
mypy in CI process.Line 268 of the doc has deepcopy:
tx = copy.deepcopy(txpool[0])
Is there more information about deepcopy?
The eth-typing package was released on PyPI recently, so we can start using it (at least Address and Hash32 and maybe BlockNumber).
current_vote can only keep track of up to 248 notaries' vote count. If committee size exceed 248(reason described here), we will need more variables to keep track of them.
james@james-Streacom:~$ git clone https://github.com/ethereum/sharding/
Cloning into 'sharding'...
remote: Counting objects: 1368, done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 1368 (delta 18), reused 40 (delta 17), pack-reused 1326
Receiving objects: 100% (1368/1368), 334.40 KiB | 198.00 KiB/s, done.
Resolving deltas: 100% (963/963), done.
james@james-Streacom:~$ virtualenv -p python3.6 sharding
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/james/sharding/bin/python3.6
Also creating executable in /home/james/sharding/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
james@james-Streacom:~$ cd sharding
james@james-Streacom:~/sharding$ python setup.py install
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
[Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-17613.write-test'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python2.7/dist-packages/
Perhaps your account does not have write access to this directory? If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account. If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.
For information on other options, you may wish to consult the
documentation at:
https://setuptools.readthedocs.io/en/latest/easy_install.html
Please make the appropriate changes for your system and try again.
The SMC has a constructor argument _LOOKAHEAD_LENGTH that corresponds to the config dict entry LOOKAHEAD_PERIODS. They should share the same name (except for the underscore) because then one can easily convert a constructor parameter dictionary to a config dict without rewriting all the names.
Also: the config object has a NOTARY_REWARD entry which seems to be unused right now.
Thanks to @vbuterin for providing some suggestions.
Get down to ~90-100k gas in practice without too much effort
ID = num256_add(shift(parent, -192), num256_and(sha3(header), as_num256(<valueof 2**192 - 1 here>)))collations_with_score mappingvalidators mapping and remove validator_code_addr:
validators: public({
# Amount of wei the validator holds
deposit: wei_value,
# The address which the validator
validation_addr: address,
# Addess to withdraw to
return_addr: address,
# The cycle number which the validator would be included after
# Will be [DEPRECATED] for stateless client
cycle: num,
}[num])get_eligible_proposer return validator_addradd_header by removing sig field:add_header(shard_id: num, expected_period_number: num, period_start_prevhash: bytes32, parent_collation_hash: bytes32, tx_list_root: bytes32, collation_coinbase: address, post_state_root: bytes32, receipt_root: bytes32, collation_number: num)assert msg.sender == self.get_eligible_proposer(shard_id, block.number / self.period_length)entire_header_hash: entire_header_hash = sha3(concat(as_bytes32(shard_id), as_bytes32(expected_period_number) ... as_bytes32(collation_number)))assert msg.sender == self.validators[validator_index].validator_addrLine 163 of the Sharding Specification is a broken link to the Ethereum Chart for "Chain Data Size Full", as it is no longer available.
Recommend mitigation measure is to just provide a link to the Ethereum Charts https://etherscan.io/charts or rephrase and provide a specific link to "ChainData Size Growth (FAST sync)" blockchain mode using the Geth Client https://etherscan.io/chart2/chaindatasizefast
Basically referring to eth_service.py and our sharding simulations to implement shard_service.py.
Synchronizer for shardingHere are some ideas for improvements of the API of SMCHandler and ShardTracker I got when implementing SMCService (most of them for the __init__ method and some probably a little bit subjective):
I think SMCHandler should be renamed to just SMC as its a subclass of web3.Contract. While it technically does not refer to the contract itself, SMC would be more inline with web3's terminology (contract <- sharding manager contract).
Both handler and tracker take a config dictionary which mainly contains parameters for the contract. I think it would be better if those parameters would be retrieved from the on-chain contract itself, otherwise it's possible that there is a mismatch. The remaining two ("DEFAULT_GAS" and "GAS_PRICE") could just be passed as standalone parameters or removed (see next bullet point).
The various send_transaction methods in SMCHandler take a gas argument. As the required gas of these functions is known (it's part of the ABI) it would be nice if it would just use that number.
ShardTracker expect a LogHandler which could just be created by the ShardTracker itself.
To create the SMCHandler one needs to provide the ABI. As the ABI doesn't change that parameter is useless, the constructor could just load the ABI from JSON.
Deploying the SMC is quite cumbersome. Two ideas here: Either make the test utils importable for other packages or, even better, make deploying easy by setting the bytecode. Then one can just use the constructor functionality provided by web3 (if I understand the docs correctly, haven't actually tried it).
@NIC619 what do you think of these? I would have already started making PRs, but wanted to discuss first, and also give you the chance to work on them if you like. I'm mainly blocked by the last point because there's not really a good way to test the SMCService right now.
There's an old simulation without stateless client: https://github.com/hwwhww/sharding/tree/old_fast_sync/simulation
So after we have finished the stateless client implementation, we can modify this simulation for
testing.
Adding vyper_compile_script.py in the repo will help everyone knows how it works faster.
vyper_compile_script.py under /tools directory.I was not sure where to ask this question, so if this is the wrong channel, feel free to delete it (in that case I would appreciate a hint, what the correct channel would be)
I read the account redesign eip and it seems a lot more streamlined than the current account representation.
My concern is with the two added address_lists and that if all the touched accounts have to be known up front then certain contracts that have a more dynamic address model, that calls other contracts based on its current (changeable) state can get in trouble as it is now possible that between the creation of the transaction and the execution of the transaction another transaction changed the state and the current adress_lists do no longer correspond to the actual set of accessed accounts.
As accounts do no longer have a key, value store this feature has to be emulated by creating a contract per entry and choosing the init code of the contract in a way, that the contracts address can be derived from the key which makes the behaviour described above even apply for contracts that access a value in a emulated key value store where the key is dependent on the state of the contract and not just on the transactions input data.
Is a variant planned where the read and write address lists can be omitted, such that contracts with the previously described dynamic behavior become possible without the risk that multiple tries become necessary?
The web3 contract class (of which SMCHandler is a subclass) provides deployment API (see https://web3py.readthedocs.io/en/stable/contracts.html#contract-deployment-example for an example). We can't use that right now as SMCHandler doesn't know about its bytecode (and ABI?) right now.
https://github.com/ethereum/wiki/wiki/Wishlist#parallelization
This link gives a 404 error: Sharding account redesign EIP: https://github.com/ethereum/sharding/blob/master/account_redesign_eip.md
I'm not sure what it is meant to refer to. @vbuterin, may you please clarify?
pip freeze):
rajat@rajat-VirtualBox:~/Desktop/sharding-master$ sudo python3 setup.py build
/home/rajat/.local/lib/python3.6/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '0.0.2-alpha.2' to '0.0.2a2'
warnings.warn(tmpl.format(**locals()))
running build
running build_py
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
file sharding.py (for module sharding) not found
running egg_info
writing sharding.egg-info/PKG-INFO
writing dependency_links to sharding.egg-info/dependency_links.txt
writing requirements to sharding.egg-info/requires.txt
writing top-level names to sharding.egg-info/top_level.txt
file sharding.py (for module sharding) not found
reading manifest file 'sharding.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sharding.egg-info/SOURCES.txt'
file sharding.py (for module sharding) not found
LogHandler.get_log uses the start of the current period if from_block if from_block is None.sharding/sharding/handler/log_handler.py
Lines 35 to 38 in 87625b6
from_block is not set in ShardTracker.get_add_header_logs. So it will always query the logs from the start of the current period even when some periods are missed.Possible solutions:
latest_block_number in ShardTracker and using it as from_block in next round.My first instinct is Proposal 1. But on second thought, it's tricky to handle maintain variable here. Avoiding keeping status in sharding repo APIs is a better call.
cc @NIC619
I've done that here: https://github.com/jamesray1/sharding.
Although you've probably realized by now, I am working on the SMC. It is available here: https://github.com/Drops-of-Diamond/sharding/blob/develop/smc/Sharding_Manager_Contract.v.py. I won't make a PR as there is still further work to do on it, but it may be delayed due to communicating with colleagues and reviewing their work in Rust at https://github.com/Drops-of-Diamond/diamond_drops. I've also been talking with the Vyper team on Gitter, they've been very helpful.
The sample function should be coded in such a way that any given validator randomly gets allocated to some number of shards every SHUFFLING_CYCLE_LENGTH, where the expected number of shards is proportional to the validator's balance.
My understanding is that each shard can have a differently sized coinbase. Because of this, I want to find how much Ether should be allocated to a validator in proportion to the ratio of the size of their deposit to the sum of all validator deposits.
ETH allocated per validator / total ETH = deposit per validator / sum of all validator deposits
ETH allocated per validator = total ETH * deposit per validator / sum of all validator deposits
However, my understanding is that a validator cannot validate a portion of a shard—they can only validate a whole shard, or not validate it at all. Thus, this exact amount of ETH calculated may not be able to be assigned to any validator, because there may not be any number of shards with a sum total coinbase equal to this amount. Therefore, there would need to be some overlap (and in fact you would want more than one validator per shard, maybe with some minimum limit of validators per shard).
So maybe instead, the number of shards per validator should at least equal to the ETH allocated per validator, while maybe each shard should have a minimum limit of validators per shard. Or alternatively, I understand that there is a minimum coinbase per shard, which can be set so that each shard doesn't have too few validators.
In summary,
n_shards / validator >= ETH allocated per validator = total ETH * deposit per validator / sum of all validator deposits
It seems the balance storage is forgotten...? But might only worthwhile to fix if there will be slashing condition for the notaries. Right now I just add it to backlogs, not a high priority.
notary_registry: should be notary_registry: {deregistered: int128, pool_index: int128, deposit: wei_value}[address].
sharding/contracts/validator_manager.v.py
Lines 37 to 40 in 38f437c
When release_notary: should return the balances instead of NOTARY_DEPOSIT:
sharding/contracts/validator_manager.v.py
Line 230 in 38f437c
For integrating with Trinity, sharding package has to use the latest packages.
pyrlp, py-evm, and eth-tester version requirements.It seems there will be a dependency issue at pyethereum ... needs bumping some related packages.
Can someone create a Gitter room and add a link to it in a badge in the readme? I would create a room myself, but it's best if someone who has access to create a repo under the Ethereum organization does it.
Also add a license for the repo and add a license badge. I suggest CC BY SA 4.0.
I've almost finished going through the MDN Javascript guide, after that I'll continue reading through this code and start building an implementation in JS.
py-evm version
shard_chain.py of shard chain is like ethereum.pow.chain class in main chainvalidator manager contract (VMC) to verify the contract, so it means we need to do the constant message call on main chain!)validator manager contract)
py-evm to test validator manager contract and the codebase
sharding.tools.tester, which is the sharding version ofethereum.tools.tester. It contains some new sharding-only functions like collate and sharding_deposit.web3.py to connect the geth main chain clientThis is for https://github.com/ethereum/sharding/blob/develop/docs/doc.md#fetch-candidate-heads-in-reverse-sorted-order.
This is just initializing the variables to the default values of their type in zero-byte representation.
unchecked_logs = []
current_checking_score = None
Maybe clarify in a comment before the function definition that unchecked_logs is added to by the CollationAdded log.
When reading this section my confusion was compounded when it said after the code:
To re-express in plain language, the idea is to scan backwards throughCollationAddedlogs (for the correct shard), and wait until you get to one whereisNewHead = True. Return that log first, then return all more recent logs with a score equal to that log withisNewHead = False, in order of oldest to most recent. Then go to the previous log withisNewHead = True(this is guaranteed to have score 1 lower than the previous NewHead), then go to all more recent blocks after it with that score, and so forth.
Also the syntax for declaring CollationAdded could use this: https://viper.readthedocs.io/en/latest/structure-of-a-contract.html#events, whereas in the doc it has:
There is also one log type:
CollationAdded(indexed uint256 shard, bytes collationHeader, bool isNewHead, uint256 score)
SMC rocks.
Rename the file name.
Old sharding spec + A minimal sharding protocol
COLLATOR_DEPOSIT -> NOTARY_DEPOSITCOMMITTEE_SIZE := 135QUORUM_SIZE := 90register_notary() returns bool: Adds an entry to notary_registry, updates the notary pool (notary_pool, notary_pool_len, etc.), locks a deposit of size NOTARY_DEPOSIT, and returns True on success. Checks:
msg.value >= NOTARY_DEPOSITnotary_registry[msg.sender] does not existderegister_notary() returns bool: Sets the deregistered period in the notary_registry entry, updates the notary pool (notary_pool, notary_pool_len, etc.), and returns True on success. Checks:
notary_registry[msg.sender] existsrelease_natory() returns bool: Removes an entry from notary_registry, releases the notary deposit, and returns True on success. Checks:
notary_registry[msg.sender] existsnotary_registry[msg.sender].deregistered != 0floor(block.number / PERIOD_LENGTH) > notary_registry[msg.sender].deregistered + NOTARY_LOCKUP_LENGTH add_header(shard_id, chunk_root, period) returns bool: anyone can call this function at anytime. The first header to get included for a given shard in a given period gets in, all others don’t. Returns True on success. This function just emits a log.
HeaderAdded log (emitted from SMC add_header function)
shard_id uint256 # pointer to shard
chunk_root bytes32 # pointer to collation body
period int128 # (current block number / PERIOD_LENGTH)
proposer_address address # the address of proposershard_id bytes1 # pointer to shard
chunk_root bytes32 # pointer to collation body
period bytes3 # (current block number / PERIOD_LENGTH)
proposer_address address # the address of proposer get_committee(shard_id, period) returns a list of addresses: use the last block hash h before this period as the seed. Selecting notary_pool[sha3(h) % notary_pool_size], notary_pool[sha3(h + 1) % notary_pool_size], .... notary_pool[sha3(h + (COMMITTEE_SIZE - 1)) % notary_pool_size] notaries from notary_pool.
submit_vote(shard_id, chunk_root, period) returns bool: Sampled notaries call this function to submit vote, returns True on success. This function just emits a log.
Vote log (emitted from SMC submit_vote function)
shard_id uint256 # pointer to shard
chunk_root bytes32 # pointer to collation body
period int128 # (current block number / PERIOD_LENGTH)
notary_address address # the address of notaryshard_id bytes1 # pointer to shard
chunk_root bytes32 # pointer to collation body
period bytes3 # (current block number / PERIOD_LENGTH)
notary_address address # the address of notarydef transact for testing on shardsgas_limit = call_valmgr(chain.state, 'get_collation_gas_limit', [])
In collator.py.
shard_protocol.py)GetShardData -> we only need to get the pivot point collationCollationBodies -> adding witnesshttps://github.com/ethereum/sharding/blob/develop/docs/doc.md#create_collation
This process has three parts. The first part can be called
GUESS_HEAD(shard_id), with pseudocode here:
# Download a single collation and check if it is valid or invalid (memoized)
validity_cache = {}
def memoized_fetch_and_verify_collation(b):
if b.hash not in validity_cache:
validity_cache[b.hash] = fetch_and_verify_collation(b)
return validity_cache[b.hash]
def main(shard_id):
head = None
while 1:
head = fetch_candidate_head(shard_id)
b = head
while 1:
if not memoized_fetch_and_verify_collation(b):
break
b = get_parent(b)
fetch_and_verify_collation(c)involves fetching the full data ofc(including witnesses) from the shard network, and verifying it.
This uses c but the preceding code uses b.
What's also confusing is that collation headers do not contain a hash member.
With stateless client, the cost of reshuffling would be reduced massively. It gives us a chance to remove shuffling cycle.
The other problem is making the whole procedure fast enough to collate the collation.
So once the validator finds he was sampled as the collator of this period:
add_header tx and broadcast the collationSince we set the period length to 5 blocks, so the add_header tx must be included in Mainnet in 75 seconds.
My system: macOS Sierra 10.12.6
In the "Environment" section of the Sharding Specification Readme it provides a link to the Pyethereum Developer Notes.
In the "Installation" section of the Pyethereum Developer Notes it states that only Python versions 2.7, 3.4, or 3.5 are supported
Since my host machine is macOS I switched to the latest supported version of Python that it mentions (i.e. Python 3.5) using pyenv
pyenv install 3.5.0; pyenv global 3.5.0; pyenv versions;But when I run the Unit Tests with pytest sharding/tests/, it gives me the exception that I need Python 3.6+ Exception: Requires python3.6+, as follows:
$ pytest sharding/tests/
================================================== test session starts ==================================================
platform darwin -- Python 3.5.0, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
rootdir: /Users/Ls/code/blockchain/clones/sharding, inifile:
plugins: cov-2.5.1, catchlog-1.2.2, timeout-1.0.0
collected 1 item / 11 errors
======================================================== ERRORS =========================================================
___________________________________ ERROR collecting sharding/tests/test_collator.py ____________________________________
sharding/tests/test_collator.py:11: in <module>
from sharding import collator
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
__________________________________ ERROR collecting sharding/tests/test_main_chain.py ___________________________________
sharding/tests/test_main_chain.py:7: in <module>
from sharding.tools import tester
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:31: in <module>
from ethereum.tools._solidity import get_solidity
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/ethereum-2.1.0-py3.5.egg/ethereum/tools/__init__.py:1: in <module>
from ethereum.tools import keys, new_statetest_utils, testutils, tester, _solidity
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/ethereum-2.1.0-py3.5.egg/ethereum/tools/tester.py:47: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
---------------------------------------------------- Captured stderr ----------------------------------------------------
Failed to import bitcoin. This is not a fatal error but does
mean that you will not be able to determine the address from
your wallet file.
__________________________ ERROR collecting sharding/tests/test_receipt_consuming_tx_utils.py ___________________________
sharding/tests/test_receipt_consuming_tx_utils.py:8: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
__________________________________ ERROR collecting sharding/tests/test_shard_chain.py __________________________________
sharding/tests/test_shard_chain.py:12: in <module>
from sharding.tools import tester
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
_______________________________ ERROR collecting sharding/tests/test_state_transition.py ________________________________
sharding/tests/test_state_transition.py:19: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
______________________________ ERROR collecting sharding/tests/test_stateless_collator.py _______________________________
sharding/tests/test_stateless_collator.py:8: in <module>
from sharding import collator
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
____________________________________ ERROR collecting sharding/tests/test_tester.py _____________________________________
sharding/tests/test_tester.py:8: in <module>
from sharding.tools import tester
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
______________________________ ERROR collecting sharding/tests/test_used_receipt_store.py _______________________________
sharding/tests/test_used_receipt_store.py:3: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
___________________________ ERROR collecting sharding/tests/test_used_receipt_store_utils.py ____________________________
sharding/tests/test_used_receipt_store_utils.py:1: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
_______________________________ ERROR collecting sharding/tests/test_validator_manager.py _______________________________
sharding/tests/test_validator_manager.py:7: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
____________________________ ERROR collecting sharding/tests/test_validator_manager_utils.py ____________________________
sharding/tests/test_validator_manager_utils.py:6: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:33: in <module>
from sharding.main_chain import MainChain
sharding/main_chain.py:23: in <module>
from sharding.shard_chain import ShardChain
sharding/shard_chain.py:25: in <module>
from sharding.collator import apply_collation
sharding/collator.py:9: in <module>
from sharding import state_transition
sharding/state_transition.py:13: in <module>
from sharding.validator_manager_utils import call_valmgr
sharding/validator_manager_utils.py:3: in <module>
from viper import compiler
<frozen importlib._bootstrap>:969: in _find_and_load
???
<frozen importlib._bootstrap>:958: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:664: in _load_unlocked
???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
???
/Users/Ls/.pyenv/versions/3.5.0/lib/python3.5/site-packages/viper-0.0.2-py3.5.egg/viper/__init__.py:5: in <module>
???
E Exception: Requires python3.6+
=================================================== warnings summary ====================================================
None
pytest-catchlog plugin has been merged into the core, please remove it from your requirements.
-- Docs: http://doc.pytest.org/en/latest/warnings.html
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================= 1 warnings, 11 error in 3.51 seconds ==========================================So to run the Unit Tests I switch to the latest version of Python (even though the link in the Readme mentions that only specific versions are supported)
pyenv install 3.6.4rc1; pyenv global 3.6.4rc1Then I try to install the relevant dependencies for the Python 3.6+ version, with guidance from the Readme, but I get the following error (with either Python 3.6.3 or 3.6.4rc1):
$ python setup.py install
...
error: ethereum 2.3.0 is installed but ethereum==2.1.0 is required by {'viper'}
And when I run the Unit Tests they appear with other errors as follows (but without any Python version error anymore)
$ pytest sharding/tests/
================================================== test session starts ==================================================
platform darwin -- Python 3.6.3, pytest-3.2.5, py-1.5.2, pluggy-0.4.0
rootdir: /Users/Ls/code/blockchain/clones/sharding, inifile:
plugins: xdist-1.20.1, pythonpath-0.7.1, mock-1.6.3, forked-0.2, hypothesis-3.38.5, flaky-3.4.0
collected 1 item / 11 errors
===Flaky Test Report===
===End Flaky Test Report===
======================================================== ERRORS =========================================================
___________________________________ ERROR collecting sharding/tests/test_collator.py ____________________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_collator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_collator.py:5: in <module>
from ethereum.transaction_queue import TransactionQueue
E ModuleNotFoundError: No module named 'ethereum.transaction_queue'
__________________________________ ERROR collecting sharding/tests/test_main_chain.py ___________________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_main_chain.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_main_chain.py:7: in <module>
from sharding.tools import tester
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
__________________________ ERROR collecting sharding/tests/test_receipt_consuming_tx_utils.py ___________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_receipt_consuming_tx_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_receipt_consuming_tx_utils.py:8: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
__________________________________ ERROR collecting sharding/tests/test_shard_chain.py __________________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_shard_chain.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_shard_chain.py:6: in <module>
from ethereum.transaction_queue import TransactionQueue
E ModuleNotFoundError: No module named 'ethereum.transaction_queue'
_______________________________ ERROR collecting sharding/tests/test_state_transition.py ________________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_state_transition.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_state_transition.py:4: in <module>
from ethereum.state import State
E ModuleNotFoundError: No module named 'ethereum.state'
______________________________ ERROR collecting sharding/tests/test_stateless_collator.py _______________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_stateless_collator.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_stateless_collator.py:6: in <module>
from ethereum.transaction_queue import TransactionQueue
E ModuleNotFoundError: No module named 'ethereum.transaction_queue'
____________________________________ ERROR collecting sharding/tests/test_tester.py _____________________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_tester.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_tester.py:8: in <module>
from sharding.tools import tester
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
______________________________ ERROR collecting sharding/tests/test_used_receipt_store.py _______________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_used_receipt_store.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_used_receipt_store.py:3: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
___________________________ ERROR collecting sharding/tests/test_used_receipt_store_utils.py ____________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_used_receipt_store_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_used_receipt_store_utils.py:1: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
_______________________________ ERROR collecting sharding/tests/test_validator_manager.py _______________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_validator_manager.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_validator_manager.py:7: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
____________________________ ERROR collecting sharding/tests/test_validator_manager_utils.py ____________________________
ImportError while importing test module '/Users/Ls/code/blockchain/clones/sharding/sharding/tests/test_validator_manager_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
sharding/tests/test_validator_manager_utils.py:6: in <module>
from sharding.tools import tester as t
sharding/tools/__init__.py:1: in <module>
from sharding.tools import tester
sharding/tools/tester.py:6: in <module>
from ethereum.utils import (
E ImportError: cannot import name 'checksum_encode'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 11 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================== 11 error in 1.22 seconds ================================================The current bug in the Travis setup reads error: Setup script exited with error in ethereum setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers; Unordered types are not allowed and a suggested fix is to upgrade pyetheruem. Is there any specific reason as to why we used a specific version of pyethereum in setup.py? If not, I can put in a PR for this issue. Thanks!
https://github.com/ethereum/sharding/blob/develop/docs/doc.md
In the first bullet point, what does purity-verified mean?
deposit(address validationCodeAddr, address returnAddr) returns uint256: adds a validator to the validator set, with the validator's size being the msg.value (ie. amount of ETH deposited) in the function call. Returns the validator index. validationCodeAddr stores the address of the validation code; the function fails if this address's code has not been purity-verified.
network_id to shard_id
PeerManger to handle the network_id mappingnetwork_id (->shards) concurrentlyA two-layer account trie inside a single-layer trie
trie.py with new_bintrie.pyState and Account classstateless_client.pyThe previous fast sync procedure for simulation is a lazy simplified approach.
Thanks to @vbuterin, the new approach applies more verifications of the recent collations and more reliable.
[+0x08: collationHash: B_32] Request for the shard chain state snapshot which contains collationHash of the required collation.
[+0x09: stateSnapshot: B, collation: Collation] Reply to GetShardData. Reply must contain the compressed state snapshot and the head collation of the given state.
GetShardData request, which indicates a collation hash collationHash as the head collation required state on shard chain. Note that the collationHash is a pivot point collation. e.g., it can be the 100th ancestor of the head collation on validator manager contract, so it's more reliable then the head collation since the pivot point collation is supposed to be finalized.PIVOT_DEPTH; default value is 100. The pivot point is head - 100 collations.GetShardData message:
collationHash exists in this shardShardData message which contains:
collationHash as head collationcollationHashShardData message:
collation.header.hash equals the record on validator manager contractcollation.header.score equals the record on validator manager contractcollationHeader.postStateRootcollation.header.score plus PIVOT_DEPTHAfter reading https://github.com/ethereum/sharding/blob/develop/docs/doc.md, I have two questions
COLLATOR_REWARD applied on the shard?ETH?def getEligibleProposer(shardId: num, period: num) -> address:
assert period >= LOOKAHEAD_PERIODS
assert (period - LOOKAHEAD_PERIODS) * PERIOD_LENGTH < block.numberShouldn't this be:
assert period <= LOOKAHEAD_PERIODS
assert (LOOKAHEAD_PERIODS - period) * PERIOD_LENGTH > block.numberThis is supported by "The function should be able to return a value for the current period or any future period up to LOOKAHEAD_PERIODS periods ahead."
ShardTrackerSMCHandler.get_collation_proposer returns a hex address, but it should be bytes according to the type hint. ShardTrackerSMCHandler.get_collation_chunk_root returns bytes, which matches the type hint but is not consistent with get_collation_proposer.
get_collation_proposer and change get_collation_chunk_root to return str as wellget_collation_proposer to return a bytes addressPersonally I'd prefer 2, because bytes is what I expected intuitively.
Noted:
post_state_root -> state_rootparent_collation_hash -> parent_hashtx_list_root -> transactions_rootSHUFFLING_CYCLE with stateless client, the syncing procedure would be executed more frequently with much lesser execution time.get_eligible_proposer function in validator manager contract
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.