ethereum / retesteth

testeth via RPC. Test run, generation by t8ntool protocol

retesteth

A test generation tool for the test fillers https://github.com/ethereum/tests/tree/develop/src
Using state transition tool t8n
More t8n spec docs: https://github.com/ethereum/go-ethereum/tree/master/cmd/evm
Execution stats: http://retesteth.ethdevops.io/
Web interface: http://retesteth.ethdevops.io/web/

Supported clients / evm's

• geth
• besu
• nimbus
• evmone
• ethereumjs
• coregeth (etc | etctranslate configs)

Docker Instructions

1. Download docker scripts:

   • Dockerfile
   • dretesteth.sh

   To use the pyspec tf generator instead of retesteth in the docker, rename the dretesteth.sh to dtf.sh.

   Note: It is best to put these files in their own directory. The reason is that any files in the same directory, including subdirectories, will be added to the docker container, slowing things down.

2. To setup the clients required, edit the args in Dockerfile script. Setup github repo and branch/commit hash to build from. Leaving an empty "" field will disable the client build in the Docker

   ARG BESU_SRC="https://github.com/hyperledger/besu.git"
   ARG PYSPECS_SRC="https://github.com/ethereum/execution-spec-tests"
   ARG ETEREUMJS_SRC="https://github.com/ethereumjs/ethereumjs-monorepo.git"
   ARG RETESTETH_SRC="https://github.com/ethereum/retesteth.git"
   ARG GETH_SRC="https://github.com/ethereum/go-ethereum.git"
   ARG NIMBUS_SRC="https://github.com/status-im/nimbus-eth1.git"
   ARG EVMONE_SRC="https://github.com/ethereum/evmone.git"
   
   # Leave empty to disable the build, can point to commit hash as well
   ARG BESU="main"
   ARG GETH="master"
   ARG NIMBUS="master"
   ARG ETHEREUMJS="master"
   ARG RETESTETH="develop"
   ARG PYSPECS="main"
   ARG EVMONE="master"
   

3. Build docker locally (if building all clients, ~10 gb space required):

   chmod +x ./dretesteth.sh
   ./dretesteth.sh build
   ./dretesteth.sh install
   

4. Edit dretesteth.sh and setup local path to the test repo so not to type --testpath.
   Navigate to the testpath folder test. And use retesteth normally (dr will be linked to dretesteth.sh after dretesteth.sh install command):

   dr test.json
   dr testFiller.json --filltests --clients besu|evmone|ethereumjs|nimbus|t8ntool(default)
   dr testfolder
   dr -t GeneralStateTest
   dr -t BlockchainTests
   dretesteth.sh --help
   

   The command will run retesteth and clients from the docker container but using local tests. Useful options (see --help):

   --statediff
   --poststate
   --vmtraceraw | --vmtrace
   --singlenet
   

   See the usage tutorial

Building locally

Basic deps

g++-11 / g++-9 required
Make sure cmake version is higher than VERSION 3.9.3, otherwise install cmake from a different source

sudo apt-cache policy cmake
sudo apt-get update
sudo apt-get install git g++ build-essential cmake

Retesteth

git clone https://github.com/ethereum/retesteth.git
cd retesteth && git checkout develop
mkdir build && cd build
cmake ..
make -j4

You should see the successful build files generation result after (cmake .. command):

Configuring done
-- Generating done
-- Build files have been written 

Test clients

Setup at least one client (default is geth). Compile required clients locally. Use Dockerfile as a hint to setup the clients or visit official page for instructions.

Retesteth is looking for client's t8n aliases in the PATH, copy or link them in the system:

cp /geth/build/bin/evm /bin/evm
cp /nimbus/tools/t8n/t8n /bin/evm_nimbus
ln -s /besu/ethereum/evmtool/build/install/evmtool/bin/evm /usr/bin/besuevm
ln -s /evmone/build/bin/evmone-t8n /usr/bin/evmone

Setup evn vars:

ETHEREUMJS_PATH env var is required for ethereumjs client
PYSPECS_PATH env var to generate .py test fillers (https://github.com/ethereum/execution-spec-tests)
ETHEREUM_TEST_PATH env to setup default path to the test repo (so not to type --testpath)

LLLC to compile LLL basic code in the test fillers

lllc compiles Lisp Like Language, an old Ethereum smart contract language that is still in use in tests.

apt-get install --yes libboost-filesystem-dev libboost-system-dev libboost-program-options-dev libboost-test-dev
git clone --depth 1 -b master https://github.com/winsvega/solidity.git /solidity
mkdir /build && cd /build
cmake /solidity -DCMAKE_BUILD_TYPE=Release -DLLL=1 && make lllc
cp /build/lllc/lllc /bin/lllc

Optionally clean the cache:

rm -rf /build /solidity /var/cache/* /root/.hunter/*  

Solidity to compile solidity and yul code in the test fillers

wget https://github.com/ethereum/solidity/releases/download/v0.8.17/solc-static-linux
cp solc-static-linux /usr/bin/solc
chmod +x /usr/bin/solc

Installing on MacOS

See https://github.com/ethereum/retesteth/blob/develop/circle.yml file as a hint.
use cmake .. -DLOCALDEPS=BOOST to disable hunter boost autoinstall to use locally installed version of BOOST (if there are issues with boost)

Contact if any question:

Telegram: @wdimitry

retesteth's Issues

Support of commandline options for ethereum clients

A client need to support following command line arguments to work with retesteth

• --ipcpath /home/user/ (geth.ipc added by the client)
• --db-path /home/user/tempdir (datadir)
• --test (tells the client to work in test mode)

Fork Name Standards discussion

Discuss enhance implement EIP proposal for generic genesis format.

Test run summary improvement

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts2 -- --clients alethIPCDebug -d 1
...
*** 1 failure is detected in the test module "Master Test Suite"
Finishing retesteth run

--------
*** TOTAL ERRORS DETECTED: 95 errors during all test execution!

1. It's confusing to see these contradicting numbers ("1 failure" vs "95 errors"), I guess there is some internal logic to it, I don't know how to better fix it.

2. Also would be nice to see the number of tests that passed or/and total number of tests that were run.

testeth via RPC requirements

!!! THIS PAGE IS OUTDATED !!!
NEW LINK: https://github.com/ethereum/retesteth/wiki/RPC-Methods

The idea is to just ask a client to init a chain with the given genesis config and then calculate a transaction and tell the result state. So we need to come up with the following test rpc methods:

0. Support of commandline options

   • --ipcpath /home/user/ (geth.ipc added by the client)
   • --db-path /home/user/tempdir (datadir)
   • --test (tells the client to work in test mode)

   Discussion: #16
   Implemented in: CPP

1. void test_setChainParams(string _config)
   init genesis block and chain configuration
   Discussion: #4 <----- More specs here
   Implemented in: CPP

2. string eth_sendRawTransaction(string _transactionRlpWithSig)
   accept a transaction for this test configuration (could be called multiple times)
   Discussion: #6 <----- More specs here
   Implemented in: CPP

3. void test_mineBlocks(int _number)
   mine a block (mine a block with the accepted transactions.) mining could be without PoW enabled. Function should return only after block has been imported or errorer. The status of a block should be returned.
   Discussion: #8 <----- More specs here
   Implemented in: CPP

4. string test_getLogHash(string _txHash)
   get pre-calculated log has of the _txHash. hash of emptyRLPList if transaction not found.
   Discussion: #7 <----- More specs here
   Implemented in: CPP

5. void test_rewindToBlock(int number)
   set chain state as of blocknumber 'number'
   Discussion: #9 <----- More specs here
   Implemented in: CPP

6. void test_modifyTimestamp(int _timestamp)
   set the timestamp for the current block
   Discussion: #10 <----- More specs here
   Implemented in: CPP

7. Export a block (rlp) by its number or hash.

8. string web3_clientVersion()
   get client version and build information to put it into the filled test
   Discussion: #13 <----- More specs here
   Implemented in: CPP, GETH, PYTHON..

9. Json::Array debug_accountRangeAt(string _blockHashOrNumber, int _txIndex, string _address, int _maxResults)
   get the list of accounts in the given block after given transaction index with provided account mask.
   Discussion: #18 <----- More specs here
   Implemented in: CPP

10. void test_importRawBlock(std::string const& _blockRLP)
    Attempt to import a block from rlp data. RLP data might be corrupted intentionally. Function should return only after block has been imported or errorer. The status of a block should be returned.
    if PoW is disabled in the chain config, a block without PoW should be accepted. Discussion: #19

The methods formats are to be decided. Perhaps new methods would be added. Lets come up with a tes RPC protocol here.

Make a release

Please make a release and publish binaries on GitHub so they can be downloaded by CI.

string eth_sendRawTransaction(string _trRLPWithSig)

string eth_sendRawTransaction(string _transactionRLP)

Accepts a transaction RLP list of the following items:

u256 nonce;
u256 gasPrice;
u256 gasLimit;
hash32 toAddress (could be empty)
u256 value
bytes data;
byte v  (in form 27 + (0|1)  and so on)
h256 r 
h256 s

and apply on top of the current chain configuration. (add it to a block which is yet not finalized)
returns the hash of the imported transaction if transaction is valid. or 0 if it's not.

alethIPCDebug config not working

It's missing forks section in config

andrei@andrei-s76-laptop ~/d/r/build> retesteth/retesteth -- --clients alethIPCDebug
Running tests using path: "/home/andrei/dev/aleth/test/jsontests/"
Running 297 test cases...
Active client configurations: 'alethIPCDebug '
Finishing retesteth run
unknown location(0): fatal error: in "LegacyTests/Constantinople/GeneralStateTests/stCallCodes": std::runtime_error: forks not found in ClientConfig  section! 
/home/andrei/dev/retesteth/retesteth/testSuites/LegacyTestsBoost.cpp(11): last checkpoint: "stCallCodes" fixture ctor

retesteth RLPx eth protocol support ?

It is possible to extend the RLPx eth protocol to support the tests that we have on hive/retesteth.
That would mean that retesteth would run as a specific test-node in the network. where you can connect to the retesteth via standard protocol and it will send you test sync data. + some control commands.

The additional commands / changes would be:

===To run tests

1. Setting up genesis via RLPx
2. Retesteth asks to reset the sync scenario and import new genesis/blocks data to sync
3. Retesteth asks for the sync result (state info. accounts data)
   *support of NoProof (do not check PoW of blocks) and NoReward (do not check PoW and do not apply mining rewards) seal engines

===To generate tests

1. Get a transaction data from the network
2. Retesteth asks the client to generate the block with imported transaction data
3. Retesteth manipulate the block timestamp (asks the client to set a specific timestamp)

Basically the same logic that is done by RPC, but with RLPx eth protocol (devP2P)

Error messages improvement

1. Client config file parsing error

> retesteth/retesteth -t GeneralStateTests -- --clients alethIPCDebug
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Running 51 test cases...
Active client configurations: 'alethIPCDebug '
unknown location(0): fatal error: in "GeneralStateTests/stCallCodes": dataobject::DataObjectException: expected ']' closing the array! around: stantinopleFix",
            "Istanbul"
}

/home/andrei/dev/retesteth/retesteth/testSuites/StateTestsBoost.cpp(81): last checkpoint: "stCallCodes" fixture ctor

Error message is confusing and should better mention that the error happened in the config file (now it looks like it's something about test file)

2. Missing fork name in client config

> retesteth/retesteth -t GeneralStateTests -- --clients alethIPCDebug
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Running 51 test cases...
Active client configurations: 'alethIPCDebug '
Running tests for config 'Ethereum aleth on IPC Debug' 2
Test Case "stCallCodes": 
24%...
48%...
72%...
96%...
100%
Error: Specified network not found: 'Istanbul' (callcallcall_000_OOGMAfter)
Error: Specified network not found: 'Istanbul' (callcodecallcodecallcode_111_SuicideMiddle)
Error: Specified network not found: 'Istanbul' (callcallcallcode_001_SuicideMiddle)
Error: Specified network not found: 'Istanbul' (callcallcall_000_SuicideEnd)

Specified network not found - it's not clear where it is not found, config file is not mentioned

3. Missing config for the fork

> retesteth/retesteth -t GeneralStateTests -- --clients alethIPCDebug
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Running 51 test cases...
Active client configurations: 'alethIPCDebug '
Finishing retesteth run
unknown location(0): fatal error: in "GeneralStateTests/stCallCodes": std::runtime_error: template .json config for network 'Istanbul' in alethIPCDebug

It's not clear at all that it looked for the config file and hasn't found it.

4. Misssing additonalForks field in the client config file

> retesteth/retesteth -t GeneralStateTests/stRevertTest -- --clients alethIPCDebug
Running 1 test case...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Finishing retesteth run
unknown location(0): fatal error: in "GeneralStateTests/stRevertTest": std::runtime_error: additonalForks not found in ClientConfig  section! 
/home/andrei/dev/retesteth/retesteth/testSuites/StateTestsBoost.cpp(119): last checkpoint: "stRevertTest" fixture ctor

*** 1 failure is detected in the test module "Master Test Suite"

It's not really clear to the user, what is ClientConfig section and where they should fix it.

General tip: always include the path to the relevant file in error messages.

5. -d/-v/-g filters out everything

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts2 -- --clients alethIPCDebug --singletest CallRipemd160_1 -d 1
...
Error: Specified filter did not run a single transaction!  (stPreCompiledContracts2/CallRipemd160_1, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)

Maybe better There are no cases in the test for the given data/value/gas filter.
Should also say what did I requested (-d 1)

6. Invalid test name

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts2 -- --clients alethIPCDebug --singletest non-existing
Running 1 test case...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Filter: 'non-existing'
unknown location(0): fatal error: in "GeneralStateTests/stPreCompiledContracts2": test::BaseEthException: Could not find a filler for provided --singletest filter: 'non-existing', or no tests were found in the test suite folder!
/home/andrei/dev/retesteth/retesteth/testSuites/StateTestsBoost.cpp(88): last checkpoint: "stPreCompiledContracts2" fixture ctor

*** 1 failure is detected in the test module "Master Test Suite"

Don't confuse the user, please, and say exactly what's missing - a filler or a test.

unhandeled exceptions

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts -- --clients alethIPCDebug --singletest modexp_37120_37111_97_1000000 --verbosity 5
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Running 1 test case...
Active client configurations: 'alethIPCDebug '
Checking test filler hashes for GeneralStateTests/stPreCompiledContracts
Filter: 'modexp_37120_37111_97_1000000'
Running tests for config 'Ethereum aleth on IPC Debug' 2
Test Case "stPreCompiledContracts": 
100%
Running modexp_37120_37111_97_1000000: 
Finishing retesteth run
terminate called after throwing an instance of 'test::BaseEthException'
  what():  ERROR OCCURED RUNNING TESTS: Error on JSON-RPC call (modexp_37120_37111_97_1000000, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0): Transaction index 1 out of range (0) for block 0x1 Request: {"jsonrpc":"2.0","method":"debug_accountRange","params":["0x1", 1, "", 20],"id":8}
Finishing retesteth run
fish: “retesteth/retesteth -t GeneralS…” terminated by signal SIGABRT (Abort)

If the clients answers with error to debug_accountRange, perhaps retesteth shouldn't crash but skip the detailed account output.

Also it's better not to crash from unhandled exceptions, but catch them somewhere at the high level (like main function), log an error and exit normally with error code.

std::string test_getBlockStatus(std::string const& _blockHash)

After the client receive a command to mine a block or it has received a block RLP to import it should be possible to get information of the status of that block:

std::string Test::test_getBlockStatus(std::string const& _blockHash)

The return status could be:

        case QueueStatus::Bad: return "Bad";
        case QueueStatus::Importing: return "Importing";
        case QueueStatus::Ready: return "Ready";
        case QueueStatus::Unknown: return "Unknown";
        case QueueStatus::UnknownParent: return "UnknownParent";

After the testsool told the client to mine a block or has send an rlp to import it should have a way to get a feedback from the client tool. Some blocks might take time to import. Block import time might be different depending on client implementation. Sometimes a tool would send invalid block rlp intentionally and it should check that the client has rejected the malicious block.

state transition tool for test generation

Suggestion:

Each client that want to be able to generate the tests or debug certain state transition implements a state-transition tool as an alternative to test-rpc methods.
The state-transition tool takes input like this (to be discussed):

{
    "genesis" : {
        "Coinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
        "Difficulty" : "0x020000",
        "GasLimit" : "0x0f4240",
        "Nonce" : "0x0000000000000000",
        "ExtraData" : "0x00",
        "Timestamp" : "0x00",
        "Mixhash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
        "Alloc" : {
            "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x600160010160005500",
                "nonce" : "0x00",
                "storage" : {
                }
            },
            "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x",
                "nonce" : "0x00",
                "storage" : {
                }
            }
        },
        "Config" : {
            "HomesteadBlock" : 0,
            "EIP150Block" : 0,
            "EIP155Block" : 0,
            "EIP158Block" : 0,
            "ByzantiumBlock" : 0,
            "ConstantinopleBlock" : 0,
            "PetersburgBlock" : 0,
            "IstanbulBlock" : 0
        }
    },
    "transactions": [
      {
        "gas": "0x5208",
        "gasPrice": "0x2",
        "hash": "0x0557bacce3375c98d806609b8d5043072f0b6a8bae45ae5a67a00d3a1a18d673",
        "input": "0x",
        "nonce": "0x0",
        "r": "0x9500e8ba27d3c33ca7764e107410f44cbd8c19794bde214d694683a7aa998cdb",
        "s": "0x7235ae07e4bd6e0206d102b1f8979d6adab280466b6a82d2208ee08951f1f600",
        "to": "0x8a8eafb1cf62bfbeb1741769dae1a9dd47996192",
        "v": "0x1b",
        "value": "0x1"
      },
      { 
        "gas": "0x5208",
        "gasPrice": "0x2",
        "hash": "0x0557bacce3375c98d806609b8d5043072f0b6a8bae45ae5a67a00d3a1a18d673",
        "input": "0x",
        "nonce": "0x0",
        "r": "0x9500e8ba27d3c33ca7764e107410f44cbd8c19794bde214d694683a7aa998cdb",
        "s": "0x7235ae07e4bd6e0206d102b1f8979d6adab280466b6a82d2208ee08951f1f600",
        "to": "0x8a8eafb1cf62bfbeb1741769dae1a9dd47996192",
        "v": "0x1b",
        "value": "0x1"
      }
  ]
}

and produces an output like this (to be discussed):

                "blockHeader" : {
                    "bloom" : "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
                    "coinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
                    "difficulty" : "0x20000",
                    "extraData" : "0x00",
                    "gasLimit" : "0xf460f",
                    "gasUsed" : "0xa034",
                    "hash" : "0xb264d43a3f23b55eb20e71c2c42676beda6086bb5b9582ba703ebe973d8da915",
                    "mixHash" : "0x0000000000000000000000000000000000000000000000000000000000000000",
                    "nonce" : "0x0000000000000000",
                    "number" : "0x1",
                    "parentHash" : "0x3a90b4fc4c29d3e867c974fa900b7c97e80776443c769892e327712c5abf4ebf",
                    "receiptTrie" : "0x5a0b89fb1e5f44d02b5900c39683a1796c591b2c3b08e3b1bfc94f5aa4f3c44d",
                    "stateRoot" : "0x23c7b48b8252701eb5a28a96a77497ad794a54a093f502b21698351d7dd01c0b",
                    "timestamp" : "0x3e8",
                    "transactionsTrie" : "0x93ca2a18d52e7c1846f7b104e2fc1e5fdc71ebe38187248f9437d39e74f43aab",
                    "uncleHash" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347"
                },
        "Post" : {
            "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x600160010160005500",
                "nonce" : "0x00",
                "storage" : {
                }
            },
            "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
                "balance" : "0x0de0b6b3a7640000",
                "code" : "0x",
                "nonce" : "0x00",
                "storage" : {
                }
            }
       }

Client config not found

Hi! I’m following these steps https://github.com/ethereum/retesteth/wiki/Creating-a-State-Test-with-retesteth (MacOS, compiled version of retesteth, TCP conn, geth) and ran into ‘Client config not found’ error.

~/Developer/retesteth/build|master ⇒  ./retesteth/retesteth -t GeneralStateTests/stExample -- --clients "gethTCP" --filltests --testpath "~/Developer/tests"
Running tests using path: "~/Developer/tests"
Running 1 test case...
Active client configurations: 'gethTCP '
Finishing retesteth run
unknown location:0: fatal error: in "GeneralStateTests/stExample": std::runtime_error: Client config not found: ~/Developer/tests/Retesteth/gethTCP/config
/Users/user/Developer/retesteth/retesteth/testSuites/StateTestsBoost.cpp:83: last checkpoint: "stExample" fixture ctor

*** 1 failure is detected in the test module "Master Test Suite"

But the config path is right. Is it possible to tell me where am I wrong?

additonalForks in client config files

1. It is misspelled (should be additionalForks)

2. Is it possible to make them optional? Currently nothing works if they're omitted.

> retesteth/retesteth -t GeneralStateTests/stRevertTest -- --clients alethIPCDebug
Running 1 test case...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Finishing retesteth run
unknown location(0): fatal error: in "GeneralStateTests/stRevertTest": std::runtime_error: additonalForks not found in ClientConfig  section! 
/home/andrei/dev/retesteth/retesteth/testSuites/StateTestsBoost.cpp(119): last checkpoint: "stRevertTest" fixture ctor

*** 1 failure is detected in the test module "Master Test Suite"

Consider getting rid of forks in config files

I don't really get what's the purpose of forks (and additonalForks) in config files.

They look all identical to me in all of the currently existing client configs (in tests repo). You could just hard-code them in retesteth (or have a single config file for retesteth), that would simplify everything for a user and reduce maintenance pain 6x.

Is it to adapt to the possibility of forks having different names in different clients? I think in reality all the clients have some adaptors in their code to integrate retesteth support, so I don't think it's really an issue (currently aleth, geth and besu all work with the same genesis config jsons, don't they?)

Is it to be able to run the subset of forks for each test? This should be better tuned with command line arguments to retesteth. No, it fails in case not all of the forks requited for tests are listed.

To summarize, it seems that having *.json config files should be enough, you don't really need to repeat their names in config file.

Feature suggestion: support individual EIP activation

This could help with adopting EIP-centric forking process and allowing EIP champions to generate tests from their reference implementation without it being merged into clients' main line.

The idea is to support the tests format where in place of a fork name you would have a string like ForkName+XXX where ForkName is one of Frontier, Homestead etc. and XXX is the number of one of the proposed (but not yet accepted) EIPs. See ethereum/tests@8bbbd48#diff-ee4ed374f5a8e138384eb7782c7b808fR39 for an example of such test.

Currently geth and aleth already support this kind of "fork name" in their testing tools.

For this to work in retesteth you would need clients to also support chain config format that includes these additional EIPs (so that retesteth could send it via setChainParams.)
Aleth doesn't yet support this, but it's not difficult to implement when needed, the main challenge is to come up with the exact chain config format for this.

Calculate nonce for the test writer in a blockchain test

Blockchain tests can contain multiple transactions. If we decide to add a transaction in the middle, we need to renumber all the nonce values, which is a pain.

It would be nice if when we don't specify a nonce retesteth would just figure it out for us. The initial nonce is already known (from the pre section), and retesteth knows exactly what transactions are being executed and at what order.

Some nitpicky suggestions around socketType and socketAddress

1. What retesteth calls "TCP"-type connection (in docs and in cofigs) all clients call more accurately "HTTP":
   geth - "HTTP-RPC" https://github.com/ethereum/go-ethereum#programmatically-interfacing-geth-nodes
   Besu - "JSON-RPC over HTTP" https://besu.hyperledger.org/en/stable/HowTo/Interact/APIs/API/
   Nethermind - "JSON RPC via HTTP" https://nethermind.readthedocs.io/en/latest/jsonrpc.html

2. socketAddress could perhaps better be called endpoint

3. endpoint could include http:// (maybe at some point you support WebSockets, then ws:// will be possible)

4. If endpoint includes http:// you don't need socketType at all, it can be determined from the path/address itself.

GCC9 build error

Building with GCC 9 fails, see ethereum/aleth#5798 for some related fixes

[ 46%] Building CXX object libdevcore/CMakeFiles/devcore.dir/SHA3.cpp.o
In file included from /home/andrei/dev/retesteth/libdevcore/SHA3.cpp:22:
/home/andrei/dev/retesteth/libdevcore/SHA3.h: In function ‘dev::SecureFixedHash<32> dev::sha3Secure(dev::bytesConstRef)’:
/home/andrei/dev/retesteth/libdevcore/SHA3.h:41:131: error: implicitly-declared ‘constexpr dev::SecureFixedHash<32>::SecureFixedHash(const dev::SecureFixedHash<32>&)’ is deprecated [-Werror=deprecated-copy]
   41 | inline SecureFixedHash<32> sha3Secure(bytesConstRef _input) { SecureFixedHash<32> ret; sha3(_input, ret.writable().ref()); return ret; }
      |                                                                                                                                   ^~~
In file included from /home/andrei/dev/retesteth/libdevcore/SHA3.h:27,
                 from /home/andrei/dev/retesteth/libdevcore/SHA3.cpp:22:
/home/andrei/dev/retesteth/libdevcore/FixedHash.h:245:22: note: because ‘dev::SecureFixedHash<32>’ has user-provided ‘dev::SecureFixedHash<T>& dev::SecureFixedHash<T>::operator=(const dev::SecureFixedHash<T>&) [with unsigned int T = 32]’
  245 |  SecureFixedHash<T>& operator=(SecureFixedHash<T> const& _c)
      |                      ^~~~~~~~
cc1plus: all warnings being treated as errors
make[2]: *** [libdevcore/CMakeFiles/devcore.dir/build.make:167: libdevcore/CMakeFiles/devcore.dir/SHA3.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:239: libdevcore/CMakeFiles/devcore.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Inconsistency in expect.storage between state and blockchain tests

When you have

expect:
   0xccc..cccc:
      storage:
         0x00: 0x01

And there is a value in a different storage cell, the test types behave differently:

• In a state test that's an error and the test fails
• In a blockchain test that is fine and the test is successful

I think it would be better if the state test behavior was followed by both test types.

Add the private key to the txs.json file in t8ntool

To be able to use t8ntool with openethereum (ethereum/tests#816), I need to take the inputs provided to me (alloc.json, txs.json, and env.json) and build out of them a test file. I have a way to fake the the post field of the test file. However, I cannot use the v, r, and s that appear in the txs.json file to reconstruct transaction.secretKey.

Can you please add that value to the txs.json file? I already ran geth's evm with this field added, and it ignored it and produced exactly the same result. If you want to verify, run this:

evm t8n --input.alloc alloc.json --input.txs txs.json --input.env env.json --state.fork YOLOv3 --output.alloc newalloc.json

alloc.json:

{
    "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
        "balance" : "0x0de0b6b3a7640000",
        "code" : "0x600160010160005500",
        "nonce" : "0x00",
        "storage" : {
        }
    },
    "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
        "balance" : "0x0de0b6b3a7640000",
        "code" : "0x",
        "nonce" : "0x00",
        "storage" : {
        }
    }
}

txs.json:

[
    {
        "input" : "0x",
        "gas" : "0x61a80",
        "gasPrice" : "0x1",
        "nonce" : "0x0",
        "to" : "0x095e7baea6a6c7c4c2dfeb977efac326af552d87",
        "value" : "0x186a0",
        "v" : "0x1c",
        "r" : "0xe94818d1f3b0c69eb37720145a5ead7fbf6f8d80139dd53953b4a782301050a3",
        "s" : "0x1fcf46908c01576715411be0857e30027d6be3250a3653f049b3ff8d74d2540c",
        "secretKey": "0x45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8"
    }
]

env.json:

{
    "currentCoinbase" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
    "currentDifficulty" : "0x020000",
    "currentGasLimit" : "0xff112233445566",
    "currentNumber" : "0x01",
    "currentTimestamp" : "0x03e8",
    "previousHash" : "0x9ea898ec26dd2ea9e0895dc06d4225fdb328c954b8f853448dfcc720daa86679",
    "blockHashes" : {
        "0" : "0x9ea898ec26dd2ea9e0895dc06d4225fdb328c954b8f853448dfcc720daa86679"
    }
}

When lllc fails, tell me where

When a test includes LLL code (which is the vast majority of them) and LLC fails we sometimes get an uninformation message. In those cases it would be useful to know what contract's compilation failed, information that retesteth should already have.

make --datadir --address <[ip:port], [ip:port]> and export config to folder functions

Error: EthGetBlockByTransaction unmarshal error: Transaction convertion error

I am currently integrating retesteth with an etc client in order to run particular groups of tests. When running ets test from BlockchainTest\ValidBlocks\bcValidBlockTest\SimpleTx.json I get the following error printed out:

Error: EthGetBlockByTransaction unmarshal error: Transaction convertion error: Error in DataObject: key: '' type: 'object' assert: count(_key) _key=v (DataObject::at) { "blockHash" : "0x71964bfcf258be241c04fbd8a83be92ef633c836209cb0746f96bc044c327db7", "blockNumber" : "0x1", "from" : "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b", "gasLimit" : "0xc350", "gasPrice" : "0xa", "hash" : "0x8cb8a30c1c8170e33e2317466a982e591e3b55aa02e353e185841bbeac0cf063", "data" : "0x", "nonce" : "0x0", "to" : "0x095e7baea6a6c7c4c2dfeb977efac326af552d87", "transactionIndex" : "0x0", "value" : "0xa" } (bcValidBlockTest/SimpleTx_Istanbul, fork: Istanbul, block: 0)

Is there something missing in the transaction object? I checked in retesteth src for required json fields and it seems all are there.

Documentation improvement: better document client's config files

The config files and genesis/<ForkName>.json files in https://github.com/ethereum/tests/tree/develop/Retesteth are not sufficiently documented, I think. There are some portions config files mentioned in https://github.com/ethereum/retesteth/wiki/Add-client-configuration-to-Retesteth but ideally I'd like to see what is the schema of these files, what must be included, what may be optionally included, what is not included, how retesteth uses these files exactly.

For example, currently it seems that these fields are required in config:

• name
• socketType
• socketAddress
• forks
• additonalForks (what is this, how is it different from forks?)

And in *.json:

• params
• accounts with a list of precompiles (this actually might be required only for aleth), but no other accounts

This kind of info would be useful in the guide "how to mare retesteth work with your client"

Missing d/v/g error vs missing test error

I find it weird and confusing, that these two kinds of errors look quite different:

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts2 -- --clients alethIPCDebug --singletest non-existing
Running 1 test case...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Filter: 'non-existing'
unknown location(0): fatal error: in "GeneralStateTests/stPreCompiledContracts2": test::BaseEthException: Could not find a filler for provided --singletest filter: 'non-existing', or no tests were found in the test suite folder!
/home/andrei/dev/retesteth/retesteth/testSuites/StateTestsBoost.cpp(88): last checkpoint: "stPreCompiledContracts2" fixture ctor

*** 1 failure is detected in the test module "Master Test Suite"

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts2 -- --clients alethIPCDebug --singletest CallRipemd160_1 -d 1
Running 1 test case...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Filter: 'CallRipemd160_1 dInd: 1'
Running tests for config 'Ethereum aleth on IPC Debug' 2
Test Case "stPreCompiledContracts2": 
100%
Error: Specified filter did not run a single transaction!  (stPreCompiledContracts2/CallRipemd160_1, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)

--------
TestOutputHelper detected 1 errors during test execution!
/home/andrei/dev/retesteth/retesteth/TestOutputHelper.cpp(150): error: in "GeneralStateTests/stPreCompiledContracts2": 

*** 1 failure is detected in the test module "Master Test Suite"
Finishing retesteth run

--------
*** TOTAL ERRORS DETECTED: 1 errors during all test execution!
--------

It looks like the first one is a critical error and stops retesteth immediately, while with the second one retesteth continues to run till the final TOTAL ERRORS DETECTED message.

Is it because with -d 1 you can run many tests, so it continues to look for other ones? (is it even useful for anyone to use d/v/g filters with multiple tests at once?)

So I think it would be better to output the final "TOTAL ERRORS DETECTED" summary in the fist case, too.

void test_modifyTimestamp(int _timestamp)

void test_modifyTimestamp(int _timestamp)

As some transactions execute TIMESTAMP opcode it is needed to set the timestamp of the current block that is being mined with that transaction so that a transaction could calculate the desired TIMESTAMP value;

The RPC call sequence is following:

session.test_modifyTimestamp(a.convert_to<size_t>());
session.test_addTransaction(tr.transaction.getData().asJson());
session.test_mineBlocks(1);

First set the timestamp of the current block. Then calculate some transaction with that timestamp.
Then finish and calculate the post state hash of a new block.

*This is basically needed to check that TIMESTAMP related tests has a determined results. But is might be not a good solution so far. A disscussion is welcome.

void test_rewindToBlock(int number)

void test_rewindToBlock(int number)

This test method should revert the state of the blockchain to a specific block number.
Basically cancelling the blocks so there would be no need to reset the chain genesis if we want to apply different transactions on top of the same genesis block.

void test_setChainParams(string _config)

The idea is to implement a simple genesis format to use in test rpc method. Each client that would like to run tests via rpc just implement this rpc methods.

The method here is the main test_setChainParams(string _config) method which would tell the client to init a test chain with the given state of accounts and genesis block config. A genesis block is a block and should have the block fields. A state is a list of accounts with it's storage and balances, nonce and code. No precompiled contracts are explicitly defined in the genesis state *(1).

The params section tells the client which mining method to use. "ProofOfWork" is a regular mining that accepts new block according to the hash calculation rules. The "ForkRules" sets which fork changes should be enabled to use on this chain. *(2). The forks are (Frontier, Homestead, ... Byzantium)

# Genesis config for running a StateTest via RPC (current version example)
{
    "sealEngine" : "NoProof",
    "params" : {
        "homesteadForkBlock" : "0x00"
    },
    "genesis" : {
        "author" : "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
        "difficulty" : "0x20000",
        "gasLimit" : "0x0f4240",
        "nonce" : "0x00",
        "extraData" : "0x00",
        "timestamp" : "0x00",
        "mixHash" : "0x00"
    },
    "accounts" : {
        "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
            "balance" : "0x0de0b6b3a7640000",
            "code" : "0x6001600101600055",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
            "balance" : "0x0de0b6b3a7640000",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        }
    }
}

We should discuss the format with other client developers and perhaps discuss what should be added here. The points *(1) and *(2) require some discussion I think.

EIP-2718 typed transactions are malformed

Running the ethereum general state tests against Besu, we are getting Invalid params responses from our json-rpc endpoint. Looking at what retesteth is generating for 2930 transactions, the txdata is getting RLP encoded, such that we get: RLP(01 + RLP(tr))

The spec for 2718, and what besu is expecting to be the json-rpc format, is simply the opaque bytes 01 + RLP(tr)

for example:
the first accessList transaction in stEIP2930/addressOpcodes encodes the raw transaction as

0xb9010b01f901070180018402625a0094cccccccccccccccccccccccccccccccccccccccc830186a0b8441a8451e600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f85bf859940000000000000000000000000000000000000101f842a00000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000060a701a0afb6e247b1c490e284053c87ab5f6b59e219d51f743f7a4d83e400782bc7e4b9a0479a268e0e0acd4de3f1e28e4fac2a6b32a4195e8dfa9d19147abe8807aa6f64

rather than:

0x01f901070180018402625a0094cccccccccccccccccccccccccccccccccccccccc830186a0b8441a8451e600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000f85bf859940000000000000000000000000000000000000101f842a00000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000060a701a0afb6e247b1c490e284053c87ab5f6b59e219d51f743f7a4d83e400782bc7e4b9a0479a268e0e0acd4de3f1e28e4fac2a6b32a4195e8dfa9d19147abe8807aa6f64

After discussing it with the other besu contributors, it was deemed that what retesteth is generating for typed transactions is malformed. I have taken a preliminary stab at changing this encoding, but it looks like the expectation that raw transactions are explicitly valid RLP encoded bytes is in several places in the codebase.

Documentation improvement: log verbosities clarification

From help output:

--verbosity <level>           Set logs verbosity. 0 - silent, 1 - only errors, 2 - informative, >2 - detailed

The output with values 1, 2, 3, 4 looks the same to me (the command I used was retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts -- --clients alethIPCDebug --singletest modexp_37120_37111_97_1000000 --verbosity 4)

It would be nice to better specify which verbosities are actually useful (like what is output at 5, what at higher values, what is max)

Bug report, `-d <n>` and `--poststate` combo doesn't work

It doesn't produce a post state.

./dretesteth.sh -t GeneralStateTests/stExample -- --filltests --testpath ~/tests --singletest add11 --poststate

Results in

Running 1 test case...
Running tests using path: /tests
Active client configurations: 't8ntool '
Filter: 'add11'�[0m
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "stExample": 
100%
PostState �[33m (stExample/add11, fork: Berlin, TrInfo: d: 0, g: 0, v: 0)�[0;31m : 
�[0mHash: 0x4cd2b714816b01fba67cb4b35328db86fa9e5fbb1ef42de81c6fe8f3c697704f
PostState �[33m (stExample/add11, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)�[0;31m : 
�[0mHash: 0xe4c855f0d0e96d48d73778772ee570c45acb7c57f87092e08fed6b2205d390f4

State Dump:�[33m (stExample/add11, fork: Berlin, TrInfo: d: 0, g: 0, v: 0)�[0;31m�[0m 
{
    "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
        "balance" : "0x0de0b6b3a76586a0",
        "code" : "0x600160010160005500",
        "nonce" : "0x00",
        "storage" : {
            "0x00" : "0x02"
        }
    },
    "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba" : {
        "balance" : "0xa868",
        "code" : "0x",
        "nonce" : "0x00",
        "storage" : {
        }
    },
    "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
        "balance" : "0x0de0b6b3a761d0f8",
        "code" : "0x",
        "nonce" : "0x01",
        "storage" : {
        }
    }
}�[0m

State Dump:�[33m (stExample/add11, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)�[0;31m�[0m 
{
    "0x095e7baea6a6c7c4c2dfeb977efac326af552d87" : {
        "balance" : "0x0de0b6b3a76586a0",
        "code" : "0x600160010160005500",
        "nonce" : "0x00",
        "storage" : {
            "0x00" : "0x02"
        }
    },
    "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba" : {
        "balance" : "0xa034",
        "code" : "0x",
        "nonce" : "0x00",
        "storage" : {
        }
    },
    "0xa94f5374fce5edbc8e2a8697c15331677e6ebf0b" : {
        "balance" : "0x0de0b6b3a761d92c",
        "code" : "0x",
        "nonce" : "0x01",
        "storage" : {
        }
    }
}�[0m
*** Total Tests Run: 2

However,

./dretesteth.sh -t GeneralStateTests/stExample -- --filltests --testpath ~/tests --singletest add11 --poststate -d 0

Results in

Running 1 test case...
Running tests using path: /tests
Active client configurations: 't8ntool '
Filter: 'add11 dInd: 0'
Running tests for config 'Ethereum GO on StateTool' 2
Test Case "stExample": 
100%
PostState  (stExample/add11, fork: Berlin, TrInfo: d: 0, g: 0, v: 0) : 
Hash: 0x4cd2b714816b01fba67cb4b35328db86fa9e5fbb1ef42de81c6fe8f3c697704f
PostState  (stExample/add11, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0) : 
Hash: 0xe4c855f0d0e96d48d73778772ee570c45acb7c57f87092e08fed6b2205d390f4
WARNING: GState transaction filter is set. Disabling generated test run!

*** No errors detected
*** Total Tests Run: 1

Documentation improvement: list available test suites

It would be nice to document what a user can provide to -t argument. There are many top-level suites now - GeneralStateTests, BCGeneralStateTests, BlockChainTests, LegacyTests It's not obvious at all what is available there, and what kind of tests are in each of these.

Also mentioning retesteth --list_content might be useful, though it outputs a lot of stuff I don't understand (unit tests?)

fork names standart (Homestead, Frontier...)

We need a document that would describe which changes were applied with those forks and how to correctly read tests with those fork names and run the transaction on those fork rules.

Currently we have. "Frontier, Homestead, EIP150, EIP158, Byzantium, Constantinople" in the test files.
Also we have FrontierToHomesteadAt5 and so on for transition rules.

build(docker): run as non-root user

retesteth doesn't require root permission to run. Dockerfile can be improved by running as non-root user.

Allow the expect: section to have information that is invisible to the evm

At present the expect section only has accounts and their attributes (balance, nonce, code, and storage). This means we can only test results that are visible to the evm.

It would be useful to have additional information available, which retesteth gets (or can get) from t8ntool. Specifically, we could use visiblity into:

1. Information from --vmtraceraw. For example, it might be useful to specify that when contract A is executed, when the program counter is B, the stack should have these values: [... , ... , ...], or that the gas spent between PC=x and PC=y is
   a certain amount.
2. receipt.logs, the log entries emitted. Right now so the only way to check what they are is to write a blockchain test and look in the hash of the block where they are emitted to see it is the expected value (see https://github.com/ethereum/tests/blob/develop/src/BlockchainTestsFiller/ValidBlocks/bcStateTests/logRevertFiller.yml). This method is very susceptible to false positives, because any other change in the block also changes the hash.
3. receipt.transactionHash, which lets us verify that everything ran as expected.

Build improvement: don't use OLD cmake policies

This will stop working in future cmake versions

retesteth/CMakeLists.txt

Lines 4 to 5 in eeb4805

	cmake_policy(SET CMP0054 OLD)
	cmake_policy(SET CMP0045 OLD)

Json::Array debug_accountRangeAt(string _blockHashOrNumber, int _txIndex, string _address, int _maxResults)

Json::Array debug_accountRangeAt(string _blockHashOrNumber, int _txIndex, string _addressHash, int _maxResults)

Get the list of accounts of size _maxResults in the given _blockHashOrNumber after given _txIndex. AddressMap contains addressHash - > address starting from given _addressHash.
nexKey field is the next addressHash (if any addresses left in the state).

The return is like

{
    "addressMap" : {
        "036014....109616b" : "a94f5374fce5edbc8e2a8697c15331677e6ebf0b",
        "0fbc62.....ba962a6" : "095e7baea6a6c7c4c2dfeb977efac326af552d87"
    },
    "nextKey" : "0000000000000000000000000000000000000000000000000000000000000000"
}

Allows to select range of accounts in given block number or hash after _txIndex.

When a blockchain test has failed transactions, tell me what transaction fails

void test_importRawBlock(std::string const& _blockRLP)

void RPCSession::test_importRawBlock(std::string const& _blockRLP)

Try importing a block from RLP.
RLP data might be intentionally corrupted.

hardcoded home directory

When I try to run retesteth locally, I see /home/wins in an error message.

/home/yh/src/retesteth/testeth/RPCSession.cpp(78): fatal error: in "GeneralStateTests/stCallCodes": Error connecting to IPC socket: /home/wins/.ethereum/geth.ipc

I think the directory is hardcoded, but it should not, eventually.

Incorrect setChainParams for tests with precompiles having non-zero balance

> retesteth/retesteth -t GeneralStateTests/stPreCompiledContracts -- --clients alethIPCDebug --singletest modexp_37120_37111_97_1000000

For this command client receives this config

{
    "params" : {
        "homesteadForkBlock" : "0x00",
        "EIP150ForkBlock" : "0x00",
        "EIP158ForkBlock" : "0x00",
        "byzantiumForkBlock" : "0x00",
        "constantinopleForkBlock" : "0x00",
        "constantinopleFixForkBlock" : "0x00",
        "istanbulForkBlock" : "0x00"
    },
    "accounts" : {
        "0x0000000000000000000000000000000000000001" : {
            "precompiled" : {
                "name" : "ecrecover",
                "linear" : {
                    "base" : 3000,
                    "word" : 0
                }
            }
        },
        "0x0000000000000000000000000000000000000002" : {
            "precompiled" : {
                "name" : "sha256",
                "linear" : {
                    "base" : 60,
                    "word" : 12
                }
            }
        },
        "0x0000000000000000000000000000000000000003" : {
            "precompiled" : {
                "name" : "sha256",
                "linear" : {
                    "base" : 600,
                    "word" : 120
                }
            }
        },
        "0x0000000000000000000000000000000000000004" : {
            "precompiled" : {
                "name" : "identity",
                "linear" : {
                    "base" : 15,
                    "word" : 3
                }
            }
        },
        "0x0000000000000000000000000000000000000005" : {
            "precompiled" : {
                "name" : "modexp"
            }
        },
        "0x0000000000000000000000000000000000000006" : {
            "precompiled" : {
                "name" : "alt_bn128_G1_add",
                "linear" : {
                    "base" : 500,
                    "word" : 0
                }
            }
        },
        "0x0000000000000000000000000000000000000007" : {
            "precompiled" : {
                "name" : "alt_bn128_G1_mul",
                "linear" : {
                    "base" : 40000,
                    "word" : 0
                }
            }
        },
        "0x0000000000000000000000000000000000000008" : {
            "precompiled" : {
                "name" : "alt_bn128_pairing_product"
            }
        },
        "0x0000000000000000000000000000000000000009" : {
            "precompiled" : {
                "name" : "blake2_compression"
            }
        },
        "0x0000000000000000000000000000000000000001" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000002" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000003" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000004" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000005" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000006" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000007" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x0000000000000000000000000000000000000008" : {
            "balance" : "0x01",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x3535353535353535353535353535353535353535" : {
            "balance" : "0x0202fa",
            "code" : "0x",
            "nonce" : "0x00",
            "storage" : {
            }
        },
        "0x82a978b3f5962a5b0957d9ee9eef472ee55b42f1" : {
            "balance" : "0x0de0b6b3a761fd06",
            "code" : "0x",
            "nonce" : "0x01",
            "storage" : {
            }
        },
        "0xc305c901078781c232a2a521c2af7980f8385ee9" : {
            "balance" : "0x00",
            "code" : "0x600035601c52740100000000000000000000000000000000000000006020526fffffffffffffffffffffffffffffffff6040527fffffffffffffffffffffffffffffffff000000000000000000000000000000016060527402540be3fffffffffffffffffffffffffdabf41c006080527ffffffffffffffffffffffffdabf41c00000000000000000000000002540be40060a0526330c8d1da600051141561012c576101856004356004013511151558576004356004013560200160043560040161014037600161034061014051610160600060056305f5e0fff11558576001610320526103206021806103808284600060046015f150505061038080516020820120905060005561038060206020820352604081510160206001820306601f820103905060208203f350005b",
            "nonce" : "0x01",
            "storage" : {
            }
        }
    },
    "sealEngine" : "NoReward",
    "genesis" : {
        "author" : "0x3535353535353535353535353535353535353535",
        "difficulty" : "0x020000",
        "gasLimit" : "0x05f5e100",
        "nonce" : "0x00",
        "extraData" : "0x00",
        "timestamp" : "0x00",
        "mixHash" : "0x00"
    }
}

accounts object here has duplicate keys e.g. 0x0000000000000000000000000000000000000001, which in case of aleth is parsed incorrectly by JSON parser (I think it's invalid JSON anyway)

void test_mineBlocks(int _number)

test_mineBlocks(int _number)

This method just accept a number and it tells the client to mine a _number blocks on top of the current one.
If there are string test_addTransaction(string _jsonTransaction) methods called in between test_mineBlocks(1) methos, then such transaction must be executed and included into the block. (if valid)

string web3_clientVersion()

string web3_clientVersion()

Get client version and build information to put it into the filled test.
The return value is string that looks like:
eth/v1.3.0/Linux/g++/Interpreter/Debug/c92bf7d8*/
< clientName >< clientVersion > ... < commitHash >

string test_getLogHash(string _txHash)

string test_getLogHash(string _txHash)

Get pre-calculated log has of the _txHash. hash of emptyRLPList if transaction not found.

:abi doesn't work with signed types

I try to use a signed integer type, such as int256, is an :abi field:

  transaction:
    data:
    - :label nop_3_3_160          :abi testNop(int256) 160

It fails with this error:

/retesteth/retesteth/TestOutputHelper.cpp(208): error: in "GeneralStateTests/VMTests/vmPerformance": 
Error: EncodeArgument: unhandled ArgSignatureType! (vmPerformance/test, step: GeneralStateTestFiller)
Error: StateTestFiller convertion error: StateTestFillerTransaction parse error: encodeAbi error: EncodeArgument: unhandled ArgSignatureType!
"transaction" : {
    "data" : [
        ":label nop_3_3_160          :abi testNop(int256) 160"
    ],
    "gasLimit" : [
        "80000000"
    ],
    "gasPrice" : "1",
    "nonce" : "0",
    "secretKey" : "45a915e4d060149eb4365960e6a7a45f334393093061116b197e3240065ff2d8",
    "to" : "cccccccccccccccccccccccccccccccccccccccc",
    "value" : [
        "0"
    ]
} (vmPerformance/test, step: GeneralStateTestFiller)

Incorrect failed test name

(running retesteth built from 85d45a9)

I have some failures running all GeneralStateTests:

> retesteth/retesteth -t GeneralStateTests -- --clients alethIPCDebug
Running 54 test cases...
Running tests using path: /home/andrei/dev/aleth/test/jsontests/
Active client configurations: 'alethIPCDebug '
Running tests for config 'Ethereum aleth on IPC Debug' 2
Test Case "stCallCodes": 
24%...
48%...
...
Running tests for config 'Ethereum aleth on IPC Debug' 2
Test Case "stRevertTest": 
24%...
48%...
73%...
97%...
100%
Error: Post hash mismatch remote: 0xc4373f01b4d302d94a2ebe1a590a61594224909570a584a02a87cbe335a21ab0, expected: 0xc38d195b2bed3762110c2ed971da8df619bc8fb58ecb25ed3edbcb388a998eb9 (stCallCodes/RevertPrecompiledTouch_noncestorage, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)
Error: Post hash mismatch remote: 0xc4373f01b4d302d94a2ebe1a590a61594224909570a584a02a87cbe335a21ab0, expected: 0x9385aac4644bf0cc69e9429a437f91a033332a7c868f1f50b36eeaf4715b48c6 (stCallCodes/RevertPrecompiledTouch_nonce, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)
Error: Post hash mismatch remote: 0xc4373f01b4d302d94a2ebe1a590a61594224909570a584a02a87cbe335a21ab0, expected: 0x15ec8dfdbc33f634b77e62d0c1e92e11c54e4f691256e705c9dfb2a05da029c8 (stCallCodes/RevertPrecompiledTouch_storage, fork: Istanbul, TrInfo: d: 0, g: 0, v: 0)
Error: Post hash mismatch remote: 0x2167e4fe7398bb0c350387609aed373b856e05b78de930b52469817e2d82c853, expected: 0x7a58df604177cfc4de44951e91af9e659e8c406e15fca4ee4950b284ceca3c8f (stCallCodes/RevertPrecompiledTouchExactOOG, fork: Istanbul, TrInfo: d: 13, g: 1, v: 0)

--------
TestOutputHelper detected 4 errors during test execution!
/home/andrei/dev/retesteth/retesteth/TestOutputHelper.cpp(150): error: in "GeneralStateTests/stRevertTest": 
...

It says stCallCodes/RevertPrecompiledTouch_noncestorage is the failed test, but really it should say stRevertTest/RevertPrecompiledTouch_noncestorage

Usability for Filler files: Put transaction data in the expect: section

When we have a number of data values in GeneralStateTests, we specify them in the transaction: field:

  transaction:
    data:
    # Type:    01 - RETURN    02 - REVERT
    # Gas

    # Enough gas for the called contract to finish
    - :abi f(uint,uint) 1 0x36
    - :abi f(uint,uint) 2 0x36
     .
     .
     .

Then, we refer to them by number in the expect: section:

  expect:

    # Calls that don't return data
    - indexes:
        data:
        - !!int 2
        - !!int 3
        - !!int 4

This means we need to go back and forth between these two sections to figure out what is happening. It also makes it difficult
to organize the data values according to a rational scheme, because when we reorder them we need to change all the references in the expect: sections.

It would be a lot easier to write tests if we could put the values directly in the expect: section:

  expect:

    # Calls that don't return data
    - indexes:
        data:
        - :abi f(uint,uint) 1 0x36
        - :abi f(uint,uint) 1 0x36

And have retesteth --filltests worry about putting them in the transaction and numbering them corrrectly. We might end up with more values (if we use the data field with different expect: sections, for example with different gas or network), but the extra storage won't be a problem.

On a related note, when a test fails we see the sequence number of the data entry. It would be nice if we could in addition / instead see the value.

docs: document difference between testeth and retesteth

From how I see from these documentations, retesteth is meant to be a next-generation testeth. I have spotted main differences so far:

1. Support concurrent execution of tests
2. not coupled with any particular Ethereum protocol implementation
3. worse performance(when using aleth client and one worker only) due to the need to start a node and communication is done via IPC (compared to simply calling a function in C++)
4. Not all tests supported yet. (TransactionTests / VMTests / RLPTests / BlockchainTests filling)
5. EVM trace rpc methods not specified yet

I think it would be great if the core developers can complete this list and document it somewhere as this is going to be useful for new users to understand.

Thanks

Add Yul support

This was requested in ethereum/tests#306 . The command is:

solc --assemble <file.yul>