cosmos / qa-integration

Cosmos QA tools

This repo contains scripts which can be used for quickly setting up a local test environment for any Cosmos based network with n number of nodes and m number of accounts. It installs go if it's not already installed on the system and also installs all the dependencies along with it.

These scripts require a few env variables to be setup in ./env file:-

For eg:-

goversion="1.18.1"
DAEMON=simd
DENOM=stake
CHAINID=test
DAEMON_HOME=${HOME}/.simd
GH_URL=https://github.com/cosmos/cosmos-sdk
CHAIN_VERSION='v0.45.4'
UPGRADE_NAME=test
UPGRADE_VERSION='v0.46.0'
NUM_TXS=50
NUM_MSGS=30
RPC="http://localhost:16657"

Note: Need not to export the env varibables using export command, These env values are fetched automatically by the scripts.

Commands:

Installation and Linting

make install-deps
make lint

Chain setup

make setup-chain
make pause-chain
make resume-chain
make stop-chain

Tests

To execute all the tests

make test-all

To execute multi-msg load test

make test-multi-msg

To execute single-msg load test

make test-single-msg

To execute query load test

make test-query-load

qa-integration's Issues

Add a short description at the beginning of each script.

Add integration tests for all the modules

Add all txs, queries tests for each module. It should also cover txs from different accounts to test

• vesting vs normal account txs
• feegrant vs normal (fees, without fees)
• authz vs normal
• gov proposals for different types of proposals (text, param change for all possible cases, software upgrades, cancel upgrades, community spends)

Modules:

• x/auth #48
• x/authz #44
• x/bank #37
• x/crisis
• x/distribution #43
• x/evidence
• x/feegrant #40
• x/gov #55
• x/mint #52
• x/params #56
• x/slashing #59
• x/staking #34
• x/upgrade #58

Reorganize network provisioning

Instead of having one script (cosmos_multinode.sh) which depends on system (ubuntu) and installation, let's break it down and have the following scripts:

• /deps/install_dependencies_ubuntu.sh
• /env.sh - list of all common environment variables we use across the scripts with documentation: DAEMO, $CHAINID, $DAEMON_HOME
• provision/localhost.sh -- will run N nodes.
• other improvements to easily re-run the network

Add stats for module tests

Currently individual module tests doesn't give any insights. Add stats to the module tests so that it gives some useful information. Something like:

Starting `x/bank` integration tests.
Completed `x/bank` module integration tests.
Total executed: 8/8 queries, 3/4 txs (successful/total)
Failed with errors [only if error]:
...errors here...

Improve the structure of the load tests

Context

In the first round we implemented simple node / network provisioning (using only the localhost) and few scripts to do query and tx tests. They are mixed together and we don't do checks.

Objectives

• Create 2 directories: provision and load_tests and organize files in the repositories.
• Clean up print lines / echos and only print if there is an error
• #8 Tracked in #12
• #9
• When something is wrong, stop the loops.
• #10
• #11 Tracked in #14

Propose a better scripting framework for qa-integrations

Today we are using set of bash scripts to automate the network setup and transactions. However it might be better to use other language like JavaScript / Python or Go, with an enterprise level library to manage automation.

Goal:

• propose a better scripting framework.

Add a new script which will "manage" (run) both query load and send load tests in parallel.

Use pre-build binaries, docker for testing

Now all works great but takes a lot of time to provision and installation. To improve the speed,

• Use hard coded keys via mnemonic such that scripts can avoid querying keyring everytime for account info. We can also do by setting account, validator addresses in env.
• Use pre-build binaries (default option via BINARY_URL, UPGRADE_BINARY_URL). If they are not set, it will build binary from source
• Use docker to avoid python installation and env setup

In the query test test two accounts, also verify that we are getting consistent results, and only print if something is wrong.

Automate the whole migration process

Dependencies:

• two binaries: app v1, app v2.

The script should easily do the following:

• setup a network of app @ v1
• prepare and execute the upgrade for app @ v2

Flow

• create a network with node app v1
• creating a software upgrade proposal
• copy new version to relevant nodes
• perform upgrade

Add GA for linting

Add an integration test to test software upgrade

Recently, upgrade handlers are added to the simapp. Create a script (shell should work too) to test the entire upgrade path.
Basically it should cover the following:

• Create a software upgrade proposal, vote
• Perform upgrade
• Test all txs, queries
• Test if the upgrade files are cleaned up (upgrade-info.json)

Add github actions

Add test summary

Currently, we are lagging information on how many integration tests are executed, how many failed and successful etc. Having a package to track these details would be very useful. Other app developers should be able to use it without making much effort in their integration tests.

Allow apps to extend/wire custom module tests

Currently, the QA integration scripts allows one to test cosmos-sdk functionality with any application. But it doesn't allow app developers to integrate their custom modules cleanly. Now either they should fork and build their own modules or write custom scripts to test both sdk functionality and their own app specific modules.

Solution:

• Create a simple script and make file definition to allow app developers to use QA scripts and wire their scripts for testing end-to-end. Something like

make test-all should call a script internally, which will execute all the sdk tests and app specific tests.

Reduce block creation time

Modify config of nodes to reduce time on block creation so we can avoid conditions like waiting for 7s.

Update the `send_load_test.sh` to send tokens back and forth and check the respective balances at the end.

Script to automate validator set change from exported genesis

We need to fully automate the following actions:

• export chain
• change validator set to the given keys
• start a new chain from the updated genesis and new validator set

This must work with any chain.

Comment load test stats on PR

Print stats of all load tests as a comment in PR once they are passed.

More genesis tools

Let's see how we can collaborate or integrate with the following tools:

• https://github.com/hyphacoop/cosmos-genesis-tinkerer
• https://github.com/hyphacoop/cosmos-ansible