paritytech / ink-waterfall Goto Github PK

If you take a look at the CI output of this test PR you can see that while the ink-waterfall pipeline succeeded no comment actually got posted on GitHub due to a "Bad credentials" error.

There's probably an outdated access token somewhere that we need to update.

We should add a regression test for use-ink/ink#364 to the waterfall.

Currently ink-waterfall tests have the following drawbacks (regarding the CI job running them):

1. A test could be easily broken by some change on the UI side, which is out of our control, while it would still block all ink! PRs because of non-green CI.

Currently e2e tests are simply being run versus currently deployed version of polkadot-js. The good effect of this is that it lets us catch any issue which ink! (and pallet_contract) users face while interacting with them on https://polkadot.js.org/apps, which is by now the most likely usage case. However, once these tests run on the CI, this some(not so rare)times leads to a PR stuck for ink!, which, again, could be caused by a bug\change on a UI side. So basically we are maintaining tests for the UI part, at the costs of longer PR delivery for our products.

Possible actions to solve this could be:

• Make UI projects run contracts e2e tests (on ink! examples) as part of CI\CD process of their deployment.
  Solely depends on @jacogr for polkadot-js, and on @statictype for contracts-ui.
• As part of the CI job run on a ink! PR, run ink-waterfall tests versus a concrete UI version, specially deployed for that particular run as part of that CI pipeline.
• Make ink-waterfall use cargo-contract for interactions with contracts. We could make this option to be used in ink! PR CI jobs.

2. Quite unsustainable test results like this one are noticed, which generally dissapear after one or a couple of restarts.

thread 'tests::erc721::erc721_operator_approvals' panicked at 'upload must have succeeded, but events contain only []', src/uis/polkadot_js.rs:470:9

or

 thread 'tests::erc1155::erc1155_approvals' panicked at 'upload must have succeeded, but events contain only []', src/uis/polkadot_js.rs:470:9=

These ones could suddenly happen on different examples on different runs, and just a restart is an only so far known way to treat this. I think we should look into it and make it more sustainable.

We are using the psutil crate to assert that a canvas node process is runnning. This happens here.

On Mac this is not supported though, resulting in a panic:

---- tests::multisig_plain::multisig_works_with_payable_transaction stdout ----
thread 'main' panicked at 'not yet implemented', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/psutil-3.2.1/src/process/sys/macos/process.rs:103:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

We should investigate using either a different crate or a different function from the psutils crate. I would try the latter first, there are a number of functions supported for Mac, just not the one we're using atm.

From the README it looks like ink-waterfall only tests the example contracts in the ink! repo. It would be nice to have ink-waterfall run on any ink! smart contracts so that third-party developers could have a way to automatically perform end-to-end tests. I think this is especially important given some of the complexities in testing certain scenarios (e.g cross contract) in the off-chain environment.

We should add tests for the trait-incrementer example added in use-ink/ink#932 to the waterfall. All possible contract interactions should be covered by the tests.

This can be done analog to the existing tests for trait-flipper.

For each CI run we already build each ink! example contract. We should utilize this information to provide insights on how the contract file sizes would change for a particular PR.

ToDo

• Persist the Wasm size of each compiled contract from master in the artifacts.
• After each example contract build on a PR compare this size to the size from the artifacts and output the difference. This output should be human-readable, in tabular format.
• If the contract size got bigger post a comment to the parent PR with the size diffs for all contracts. We don't want to make CI fail if contracts get bigger, since there might be features where the trade-off makes sense. So for now it's just for informational purposes.

Currently the polkadot.js UI interaction does not work, the main issue is OutOfGas from calling contracts.

This is likely because the polkadot.js UI frontend does not currently support WeightsV2. See polkadot-js/apps#8364.

The fix would be to adapt the apps frontend to allow using the proofSize returned from the call dry run, and optionally allowing the user to specify the proof size in the UI.

Our master CI currently fails due to:

…
$ echo "$REDIS_SIZES_KEY will be compared to $REDIS_SIZES_KEY_MASTER"
ink-waterfall::ink::bug/doesnt-revert-on-error::sizes will be compared to ink-waterfall::ink::master::sizes
$ redis-cli -u $GITLAB_REDIS_URI --raw lrange $REDIS_SIZES_KEY 0 -1 | tee $REDIS_SIZES_KEY.csv
tee: 'ink-waterfall::ink::bug/doesnt-revert-on-error::sizes.csv': No such file or directory
contract-terminate, 19.86, 1.532
…

Full error log: https://gitlab.parity.io/parity/ink-waterfall/-/jobs/1191960.

cc @HCastano

We should add a regression test for use-ink/ink#364 to the waterfall.