GithubHelp home page GithubHelp logo

acalanetwork / chopsticks Goto Github PK

View Code? Open in Web Editor NEW
126.0 5.0 74.0 20.15 MB

Create parallel reality of your Substrate network.

License: Apache License 2.0

JavaScript 1.09% Shell 0.56% Rust 10.28% TypeScript 86.04% HTML 1.99% CSS 0.04%

chopsticks's Introduction

Chopsticks

chopsticks logo

Create parallel reality of your Substrate network.

Quick Start

Fork Acala mainnet: npx @acala-network/chopsticks@latest --endpoint=wss://acala-rpc-2.aca-api.network/ws

It is recommended to use config file. You can check configs for examples.

Run node using config file

# npx @acala-network/chopsticks@latest --config= url | path | config_file_name
# i.e: using configs/acala.yml

npx @acala-network/chopsticks@latest -c acala

Wiki

Documentation and tutorials are available at wiki.

EVM+ tracing

Documentation for EVM+ tracing is available at EVM+ tracing.

Web testing

Run Chopsticks in browser? Now you can turn a mainnet into a devnet and play with it directly in your browser!

An example is available at acalanetwork.github.io/chopsticks, and the corresponding code can be found in web-test.

Environment Variables

For chopsticks CLI, you can find the full list of available environment variables here.

Install

Make sure you have setup Rust environment (>= 1.64).

  • Clone repository with submodules (smoldot)
    • git clone --recurse-submodules https://github.com/AcalaNetwork/chopsticks.git && cd chopsticks
  • Install deps
    • yarn
  • Build wasm. Please do not use IDE's built-in tools to build wasm.
    • yarn build-wasm

Run

  • Replay latest block
    • npx @acala-network/chopsticks@latest run-block --endpoint=wss://acala-rpc-2.aca-api.network/ws
    • This will replay the last block and print out the changed storages
    • Use option -b|--block to replay certain block hash
    • Use option --output-path=<file_path> to print out JSON file
    • Use option --html to generate storage diff preview (add --open to automatically open file)

Dry-run

  • Dry run help:
npx @acala-network/chopsticks@latest dry-run --help
  • Dry run extrinsic, same as run-block, example:
npx @acala-network/chopsticks@latest dry-run --config=configs/mandala.yml --html --open --extrinsic=0x39028400d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d01183abac17ff331f8b65dbeddd27f014dedd892020cfdc6c40b574f6930f8cf391bde95997ae2edc5b1192a4036ea97804956c4b5497175c8d68b630301685889450200000a00008eaf04151687736326c9fea17e25fc5287613693c912909cb226aa4794f26a480284d717
  • Dry run call, make sure mock-signature-host: true to fake caller's signature:
npx @acala-network/chopsticks@latest dry-run --config=configs/mandala.yml --html --open --extrinsic=0xff00000080969800 --address=5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY --at=<block_hash_optional>
  • Dry run a preimage:
npx @acala-network/chopsticks@latest dry-run --endpoint=wss://polkadot-rpc.dwellir.com --preimage=<preimage> --open
  • Dry run a preimage and execute an extrinsic after that:
npx @acala-network/chopsticks@latest dry-run --endpoint=wss://polkadot-rpc.dwellir.com --preimage=<preimage> --extrinsic=<extrinsic> --open
  • Dry run a preimage and execute a call after that, make sure mock-signature-host: true to fake caller's signature:
npx @acala-network/chopsticks@latest dry-run --config=configs/mandala.yml --preimage=<preimage> --extrinsic=<call> --address=<who> --open
  • Run a test node

    • npx @acala-network/chopsticks@latest --endpoint=wss://acala-rpc-2.aca-api.network/ws
    • You have a test node running at ws://localhost:8000
    • You can use Polkadot.js Apps to connect to this node
    • Submit any transaction to produce a new block in the in parallel reality
    • (Optional) Pre-define/override storage using option -s|--import-storage=storage.[json/yaml]. See example storage below.
    {
      "Sudo": {
        "Key": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
      },
      "TechnicalCommittee": {
        "Members": ["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"]
      },
      "Tokens": {
        "Accounts": [
          [
            ["5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY", { "token": "KAR" }],
            {
              "free": 1000000000000000,
            }
          ]
        ]
      },
      "Whitelist": {
        "WhitelistedCall": [
          [
            ["0x3146d2141cdb95de80488d6cecbb5d7577dd59069efc366cb1be7fe64f02e62c"],
            "0x" // please use 0x for null values
          ],
        ]
      }
    }
  • Run Kusama fork

    • Edit configs/kusama.yml if needed. (e.g. update the block number)
    • npx @acala-network/chopsticks@latest --config=configs/kusama.yml
  • Setup XCM multichain NOTE: You can also connect multiple parachains without a relaychain

npx @acala-network/chopsticks@latest xcm -r kusama -p karura -p statemine

Proxy

Chopsticks respect http_proxy and https_proxy environment variables. Export ROARR_LOG=true environment variable to enable log printing to stdout. To learn more, see https://www.npmjs.com/package/global-agent?activeTab=readme

Plugins

Chopsticks is designed to be extensible. You can write your own plugin to extend Chopsticks' functionality.

There are 2 types of plugins: cli and rpc. cli plugins are used to extend Chopsticks' CLI, while rpc plugins are used to extend Chopsticks' RPC.

To create a new plugin, you could check out the run-block plugin as an example.

RPC Methods

Chopsticks allows you to load your extended rpc methods by adding the cli argument --unsafe-rpc-methods=<file path>or -ur=<file path>.

WARNING:

It loads an unverified scripts, making it unsafe. Ensure you load a trusted script.

example:

npx @acala-network/chopsticks@latest --unsafe-rpc-methods=rpc-methods-scripts.js

scripts example of rpc-methods-scripts:

return {
  async testdev_testRpcMethod1(context, params) {
    console.log('testdev_testRpcMethod 1', params)
    return { methods: 1, params }
  },
  async testdev_testRpcMethod2(context, params) {
    console.log('testdev_testRpcMethod 2', params)
    return { methods: 2, params }
  },
}

Testing big migrations

When testing migrations with lots of keys, you may want to fetch and cache some storages.

There are two ways to fetch storages.

The first way is to use a config file with a prefetch-storages section:

prefetch-storages:
  - '0x123456' # fetch all storages with this prefix
  - Balances # fetch all storages under Balances pallet
  - Tokens.Accounts # fetch all storages under Tokens.Accounts stroage
  - System: Account # fetch all storages under System.Account stroage
  - Tokens:
      Accounts: [5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY] # fetch all storages for Tokens.Accounts(Alice)
  - Tokens.Accounts: [5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, { token: DOT }] # fetch this particular storage

When you starts chopsticks, it will fetch these storages in background.

Please note that only the formats mentioned above are supported for config files.

The second way is use fetch-storages subcommand to only fetch and cache storages:

npx @acala-network/chopsticks@latest fetch-storages 0x123456 Balances Tokens.Accounts
	--endpoint=wss://acala-rpc-0.aca-api.network
	--block=<blockhash> # default to latest block
	--db=acala.sqlite

The subcommand arguments could be:

  • hex: fetch all storages with this prefix
  • PalletName: fetch all storages for this pallet
  • PalletName.StorageName: fetch all storages for this storage

Please note that for both ways, fetched storages will be saved in the sqlite file specified by --db option (db: ./acala.sqlite in a config file), if not provided, it will default to ./db-{network}-{block}.sqlite.

Try-Runtime CLI

Documentation can be found here

chopsticks's People

Contributors

albertov19 avatar brenzi avatar chralt98 avatar crystalin avatar dependabot[bot] avatar dzmitry-lahoda avatar ermalkaleci avatar girazoki avatar gluneau avatar goncer avatar h4x3rotab avatar jboetticher avatar jelliedowl avatar leonardocustodio avatar misaka-0x447f avatar niall-watr avatar ntduan avatar nzt48 avatar qiweiii avatar sander2 avatar seadanda avatar shunjizhan avatar spazcoin avatar vanderian avatar vuittont60 avatar wischli avatar wsteth avatar xlc avatar zktony avatar zqhxuyuan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

chopsticks's Issues

Request: Support for rolling back

Having the possibility to resume operation at a previous block could be a great feature (for testing mostly).
Ex:

  • You generate few blocks to achieve a given state s0
  • You run 1 test specific to that state
  • you rollback to s0
  • You run another test
  • and ...

put/get for `0x` doesn't seem to work as expected

In our randomness inherent, we put an empty value to a storage item (used to ensure the inherent was set)
(see https://github.com/PureStake/moonbeam/blob/master/pallets/randomness/src/lib.rs#L258 for put and https://github.com/PureStake/moonbeam/blob/master/pallets/randomness/src/lib.rs#L306 for the check)

The logs are showing the value being set:

    4: [
      "0x7a414cb008e0e61e46722aa60abdd672e071d663aecfe81953e36656ddea98c6",
      "0x"
    ]

and retrieved later:

[11:30:41.067] TRACE (rpc-exec/23305): exec_storageGet
    blockHash: "0xb535900000000000000000000000000000000000000000000000000000000000"
    key: "0x7a414cb008e0e61e46722aa60abdd672e071d663aecfe81953e36656ddea98c6"
    value: "0x"

Howerver the runtime still crashes with

[ERROR] 11:30:41 Error: Error while executing Wasm VM: Trap: Unreachable
"panicked at 'Mandatory randomness inherent not included; InherentIncluded storage item is empty', /build/pallets/randomness/src/lib.rs:305:13"

configure inherents

have an inherent registry and allow chain config to specify what inherents to be used

Missing validationData when producing block

I've tried chopstick with Moonriver and it is not able to produce a block after sending a Tx:
yarn start dev --endpoint=wss://moonriver.kaki.dev

[15:10:29.155] INFO (txpool/4523): Building block
    hash: "0x6d31aa900f8334cbb623ab538ee633337bb7c2e3b4a1340c4f76678ddbc366d7"
    number: 3202514
    extrinsicsCount: 1
    tempHash: "0x5d42540000000000000000000000000000000000000000000000000000000000"
    timeValue: 1671199810076
    expectedSlot: 835599905038
/home/hello/projects/chopsticks/src/blockchain/inherents.ts:123
        extrinsic.validationData.relayParentStorageRoot as HexString,
                                 ^
TypeError: Cannot read properties of undefined (reading 'relayParentStorageRoot')
    at SetValidationData.createInherents (/home/hello/projects/chopsticks/src/blockchain/inherents.ts:123:34)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async InherentProviders.createInherents (/home/hello/projects/chopsticks/src/blockchain/inherents.ts:71:19)
    at async TxPool.#buildBlock (/home/hello/projects/chopsticks/src/blockchain/txpool.ts:125:23)
    at async TxPool.buildBlock (/home/hello/projects/chopsticks/src/blockchain/txpool.ts:62:5)

Question: Is it possible to Time-Travel?

I am trying out Chopsticks to implement some testing on the subquery project. I was able to start the network using chopstick but , wanted to move forward to specify era . Example scenario , Unbond staking will take 28era . How do i fast-track 28days and test the Withdrawn event.

Please do let me know if it is possible?

Library mode

Make this suitable as a library.

Use cases:

  • e2e tests
  • Integrate with other tools such block indexer
  • Run in browser dapps
  • Run in browser extensions

Need to refactor the files to have them on different packages and ensure the API are good.

refactor decode-key

  • add tests
  • improve setup to avoid unneeded work
  • cache key prefix in sqlite db
  • interactive mode - when no key provided, listen to stdin for keys

Fix Kusama block building

The consensus digest isn't exactly right.
The 2nd new block building for some reason takes forever. It is iterating all the offensive.reports and some other big storage maps and takes very long time to down all of those storages one by one.

RPC to override storage

  • RPC: dev_setStorages
  • Params:
    • values: map
      • key: hex string
      • value: hex string or null to delete
    • blockHash: hex string or null for latest block

fast forward

RPC to fast forward to a future block
basically calling create new block in a loop

improve block building performance

  • bulk storage fetch
  • For Kusama / Polkadot improve session change block handling somehow to avoid session related work, which reads a lot of storages

instructions do not work

Tried the README examples without success

> yarn install
➤ YN0000: ┌ Resolution step
➤ YN0002: │ ts-node-dev@npm:2.0.0 [cb6f5] doesn't provide @types/node (pcc22c), requested by ts-node
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 3s 398ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ yargs@npm:17.6.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yargs@npm:17.6.2 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yn@npm:3.1.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ yocto-queue@npm:0.1.0 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ zod@npm:3.19.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed in 34s 257ms
➤ YN0000: ┌ Link step
➤ YN0007: │ wasm-pack@npm:0.10.3 must be built because it never has been before or the last one failed
➤ YN0007: │ sqlite3@npm:5.1.2 [cb6f5] must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.15.12 must be built because it never has been before or the last one failed
➤ YN0007: │ bufferutil@npm:4.0.7 must be built because it never has been before or the last one failed
➤ YN0007: │ es5-ext@npm:0.10.62 must be built because it never has been before or the last one failed
➤ YN0007: │ utf-8-validate@npm:5.0.10 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 12s 142ms
➤ YN0000: Done with warnings in 49s 883ms

> yarn start dev --config=configs/kusama.yml
Error: Cannot find module '../executor/pkg'
Require stack:
- /home/abrenzikofer/chopsticks/src/executor.ts
- /home/abrenzikofer/chopsticks/src/blockchain/block.ts
- /home/abrenzikofer/chopsticks/src/blockchain/index.ts
- /home/abrenzikofer/chopsticks/src/index.ts
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/home/abrenzikofer/chopsticks/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/home/abrenzikofer/chopsticks/src/executor.ts:7:1)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Module.m._compile (/home/abrenzikofer/chopsticks/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Object.require.extensions.<computed> [as .ts] (/home/abrenzikofer/chopsticks/node_modules/ts-node/src/index.ts:1621:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/abrenzikofer/chopsticks/src/executor.ts',
    '/home/abrenzikofer/chopsticks/src/blockchain/block.ts',
    '/home/abrenzikofer/chopsticks/src/blockchain/index.ts',
    '/home/abrenzikofer/chopsticks/src/index.ts'
  ]
}

Prompt responder

Some commands prompt for input. For example sudo ... can ask for a root password.
How can I provide it through chopsticks? Is there any helper?

BTW: great library! I have few simple scripts in Fabric and moving them to Chopsticks, as it's easier to orchestrate tasks on remote.

Request: Support timed based sealing

Like the dev service, it would be great to support time based sealing like --sealing=2000 which would try to produce a block 2000ms after the previous one (every 2s basically but if the block production is slow; it might be more)

Support multiple networks

We want to run relaychain + multiple parachains at same time and connect the XCM channel to allow test XCM related code

Depends on #15

Followup of #32

Followup of #32

  • Support yaml instead of json
  • Improve error handling error message
  • Update the RPC to also accept the string format
  • Add some tests

Add sudo to a non-sudo chain

What would it take for this tool to add sudo privilleges even if the sudo pallet isn't included in the chain we are forking? The approach used by fork-off-substrate was to download the current state as a chainspec and start a dev chain with it. In our case fork-off-substrate did not produce blocks so wasn't very useful.

import config only works in `chopsticks` folder

npx version works in chopsticks folder

cd chopsticks/
npx @acala-network/chopsticks@latest dev --config=configs/karura.yml

[17:31:13.042] INFO (rpc/98663): Karura RPC listening on port 8000

however it broke in other folders

cd ../
npx @acala-network/chopsticks@latest dev --config=chopsticks/configs/karura.yml

...
Error: Error: missing field `blockHash`
    at module.exports.__wbindgen_error_new (/Users/shunjizhan/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-executor/chopsticks_executor.js:326:17)
...

if don't specify --config it works in other folders though

support fellowship

The config file can override council and technical committee members. Would be great if the kusama config could be extended to override the fellowship.
I'd like to test the whitelisted_caller flow for gov2

Inccorectly handling request ending/closure

I didn't have time to dig into it yet.

When sending a tx to the node (with jsonrpc tools or polkadotjs) the clients usually stay hanging even after the block is produced with the tx.

Ex: npx jaysonic -c ws -i localhost -p 8000 --method dev_newBlock --params '[]' =>

{
  "jsonrpc": "2.0",
  "error": {
    "code": -32000,
    "message": "Request Timeout"
  },
  "id": 1
}

`Object.hasOwn` is not a function

Getting this error on node v14.17.6, when I connect to the node with polkadot-apps:

(node:1360079) UnhandledPromiseRejectionWarning: TypeError: Object.hasOwn is not a function
    at WebSocket.<anonymous> (/.../chopsticks/src/server.ts:88:27)
    at WebSocket.emit (events.js:400:28)
    at Receiver.receiverOnMessage (/.../chopsticks/node_modules/ws/lib/websocket.js:1178:20)
    at Receiver.emit (events.js:400:28)
    at Receiver.dataMessage (/.../chopsticks/node_modules/ws/lib/receiver.js:528:14)
    at Receiver.getData (/.../chopsticks/node_modules/ws/lib/receiver.js:446:17)
    at Receiver.startLoop (/.../chopsticks/node_modules/ws/lib/receiver.js:148:22)
    at Receiver._write (/.../chopsticks/node_modules/ws/lib/receiver.js:83:10)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at Receiver.Writable.write (internal/streams/writable.js:303:10)

The command I ran is yarn dev:acala but I get this error regardles of how I start the node. Removing the !Object.hasOwn(...) check here fixes it for me:

if (!req || !Object.hasOwn(req, 'id') || !req.method) {

kusama example fails. js/apps can't connect

> yarn start dev --config=configs/kusama.yml
[12:06:18.867] INFO (6524): Args
    endpoint: "wss://kusama-rpc.dwellir.com"
    block: 15571512
    import-storage: {
      "System": {
        "Account": [
          [
            [
              "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
            ],
            {
              "data": {
                "free": "10000000000000000000"
              }
            }
          ]
        ]
      },
      "Council": {
        "Members": [
          "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
        ]
      },
      "PhragmenElection": {
        "Members": [
          {
            "who": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
            "stake": 1000000000000,
            "deposit": 1000000000000
          }
        ]
      },
      "TechnicalCommittee": {
        "Members": [
          "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
        ]
      },
      "TechnicalMembership": {
        "Members": [
          "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY"
        ]
      }
    }
    mock-signature-host: true
    db: "./db.sqlite"
    _: [
      "dev"
    ]
    config: "configs/kusama.yml"
    $0: "node_modules/ts-node/dist/bin.js"
    blockHash: "0x5c6d7556c1e0905f7ae6cce5b3ffa11b894727bb46cae724d27645a96b6deb9d"
[12:06:21.225] INFO (task/6524): Mock signature host enabled

ws://localhost:8000 stays unresponsive, although

> netstat -tulpen
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.1:40783         0.0.0.0:*               LISTEN      1000       22887      5479/node
tcp6       0      0 :::8000                 :::*                    LISTEN      1000       20878      5933/node

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.