r0gue-io / pop-cli

An all-in-one tool for Polkadot development.

Pop CLI

An all-in-one tool for Polkadot development.

Install

You can install Pop CLI as follows:

cargo install --locked --git https://github.com/r0gue-io/pop-cli

ℹ️ For detailed instructions on how to install Pop CLI, please refer to our documentation: https://learn.onpop.io/v/contracts/welcome/installing-pop-cli

A crates.io version will be available soon!

Telemetry

Pop CLI collects anonymous usage metrics to help us understand how the tool is being used and how we can improve it. We do not collect any personal information. If you wish to disable telemetry or read more about our telemetry practices please see our telemetry documentation.

Getting Started

Parachains

Use pop to create a new Parachain project. To be guided through the entire parachain creation process, simply execute

pop new parachain

If no guidance is needed, proceed with:

# Create a minimal parachain
pop new parachain my-app

pop-cli supports diverse project templates, to use a specific one use the flag --template:

# Create an assets parachain
pop new parachain my-app pop -t assets
# Create a contracts parachain
pop new parachain my-app pop -t contracts
# Create a evm parachain
pop new parachain my-app pop -t evm

We also integrate other provider templates in the tool, check them running:

pop new parachain --help

Some examples are:

# Get Parity's pallet-contracts enabled parachain template
pop new parachain my-app parity -t cpt
# Get Parity's evm compatible parachain template
pop new parachain my-app parity -t fpt

For Pop templates you can also customize your parachain by providing config options for token symbol (as it appears in chain metadata), token decimals, and the initial endowment for developer accounts. Here's how:

# Create a minimal parachain with "DOT" as token symbol, 6 token decimals and 1 billion tokens per dev account
pop new parachain my-app --symbol DOT --decimals 6 --endowment 1_000_000_000

There's also the shorter version:

pop new parachain my-app -s DOT -d 6 -i 1_000_000_000

Use pop to build your Parachain:

# Build your parachain
pop build parachain -p ./my-app

or

cd my-app
pop build parachain

Finally, to build your Parachain:

cd my-app
pop build parachain --release

Spawn Network using Zombienet

You can spawn a local network using zombienet as follows:

pop up parachain -f ./tests/zombienet.toml -p https://github.com/r0gue-io/pop-node

ℹ️ Pop CLI will automatically source the necessary polkadot binaries. Currently, these will be built if on a non-linux system.

Run a command after the network has been spun up

The following will spin up the network locally according the the zombienet file and once the network is up, it will run the command specified in --cmd:

pop up parachain -f ./tests/zombienet.toml -p https://github.com/r0gue-io/pop-node --cmd ./path/to/my/script

Contracts

Use pop to create a new Smart Contract project:

# Create a minimal Smart Contract
pop new contract my_contract

Test the Smart Contract:

# Test an existing Smart Contract
pop test contract -p ./my_contract

Build the Smart Contract:

# Build an existing Smart Contract
pop build contract -p ./my_contract

By default the contract is compiled with debug functionality included.

This enables the contract to output debug messages, but increases the contract size and the amount of gas used.

For production builds, use the --release flag: --release:

pop build contract -p ./my_contract --release

Deploy and instantiate the Smart Contract:

pop up contract -p ./my_contract --constructor new --args "false" --suri //Alice

ℹ️ If you don't specify a live chain, pop will automatically spawn a local node for testing purposes.

Some of the options available are:

• Specify the contract constructor to use, which in this example is new().
• Specify the argument (args) to the constructor, which in this example is false.
• Specify the account uploading and instantiating the contract with --suri, which in this example is the default development account of //Alice. For other accounts, the actual secret key must be provided e.g. an 0x prefixed 64 bit hex string, or the seed phrase.

⚠️ Use only for development: Use a safer method of signing here before using this feature with production projects. We will be looking to provide alternative solutions in the future!

• You also can specify the url of your node with --url ws://your-endpoint, by default it is using ws://localhost:9944.

For more information about the options, check cargo-contract documentation

Interacting with the Smart Contract:

1. Read-only Operations: For operations that only require reading from the blockchain state. This approach does not require to submit an extrinsic. Example using the get() message:

pop call contract -p ./my_contract --contract $INSTANTIATED_CONTRACT_ADDRESS --message get --suri //Alice

2. State-modifying Operations: For operations that change a storage value, thus altering the blockchain state. Include the x / --execute flag to submit an extrinsic on-chain.

Example executing the flip() message:

pop call contract -p ./my_contract --contract $INSTANTIATED_CONTRACT_ADDRESS --message flip --suri //Alice -x

E2E testing

For end-to-end testing you will need to have a Substrate node with pallet contracts. You do not need to run it in the background since the node is started for each test independently. To install the latest version:

cargo install contracts-node --git https://github.com/paritytech/substrate-contracts-node.git

If you want to run any other node with pallet-contracts you need to change CONTRACTS_NODE environment variable:

export CONTRACTS_NODE="YOUR_CONTRACTS_NODE_PATH"

Run e2e testing on the Smart Contract:

# Run e2e tests for an existing smart contract
 pop test contract  -p ./my_contract --features e2e-tests

Pallets

To create a new Pallet, simply run pop new pallet. You will have a new pallet ready for hacking. To customize the new Pallet you can follow these options:

# create a pallet with name `pallet-awesome` in the current working directory
pop new pallet pallet-awesome
# or with options
pop new pallet pallet-awesome --authors Me --description "This pallet oozes awesomeness" --path my_app/pallets

Building Pop CLI locally

Build the tool locally with all the features:

cargo build --all-features

Build the tool only for Parachain functionality:

cargo build --features parachain

Build the tool only for Smart Contracts functionality:

cargo build --features contract

Testing Pop CLI

To test the tool locally. Due to the time it can take to build a Parachain or a Smart Contract, some tests have been separated from the normal testing flow into integration tests.

Run the unit tests only:

cargo test --lib

To run the integration tests relating to Smart Contracts:

cargo test --test contract

To run the integration tests relating to Parachains:

cargo test --test parachain

Run all tests (unit + integration):

cargo test

Acknowledgements

Pop CLI would not be possible without these awesome crates!

• Local network deployment powered by zombienet-sdk
• cargo contract a setup and deployment tool for developing Wasm based Smart Contracts via ink!

License

The entire code within this repository is licensed under the GPLv3.

Please contact us if you have questions about the licensing of our products.

pop-cli's Issues

`pop new pallet`: in folder pallets

Currently when adding pop new pallet it just add it in the directory you are located by default.

A cool feature would be to detect if you are running this inside a node folder, and ask the user if wants to create it inside a pallet folder.
If yes:

1. create a pallet folder
2. create the new pallet in this folder
3. add the workspace pallet code in the root directory Cargo.toml:

[workspace]
members = [
	"node",
	"runtime",
         "pallets/*",
]

If this line is not added into Cargo.toml it will failled to build the pallet:

error: current package believes it's in a workspace when it's not:
current:   /myparachain/pallet-template/Cargo.toml
workspace: /myparachain/Cargo.toml

this may be fixable by adding `pallet-template` to the `workspace.members` array of the manifest located at: /mytest/Cargo.toml
Alternatively, to keep it out of the workspace, add the package to the `workspace.exclude` array, or add an empty `[workspace]` table to the package's manifest.

My concern here is if the user only want to create a pallet template, not adding it into a node folder. Would this question make sense to them?

Feature: Contracts `pop test e2e`

Add the feature to tests a ink! smart contract by executing all integration tests.

Follow the ink guide: https://use.ink/basics/contract-testing/#end-to-end-e2e-tests

Feature: Cache management

pop up parachain currently sources polkadot/system parachain binaries and caches them for subsequent runs. The user should be able to easily see and clear what is in the cache.

Note that this was already implemented in a previous version of the tool, so I will just port it over. We just need to align on the command structure which can allow reuse for other related functionality - e.g. inspect: pop inspect cache, clear: pop clear cache

UX: `pop build parachain` to support multiple parachains

pop new parachain my-good-chain

pop new parachain my-better-chain

The issue is when I run:

pop build parachain

IIUC, there is no way to specify which chain I want to build.

Also, the message will show:

┌   Pop CLI : Building a parachain

The message should have the parachain name.

Feature: Contract Build Output

Using cargo-contract I get the following when building:

Your contract artifacts are ready. You can find them in:
/Users/frank/dev/pop-node/contracts/pop-api-examples/balance-transfer/target/ink

  - pop_api_extension_demo.contract (code + metadata)
  - pop_api_extension_demo.wasm (the contract's code)
  - pop_api_extension_demo.json (the contract's metadata)

Whereas with pop I only get:

●  The smart contract has been successfully built.

It would be nice to including something similar, perhaps with file sizes of the resulting artifacts to add something new.

`pop build parachain`: Generate a chain specification file

Add an option for pop build parachain to generate a chain spec file after build it.

The steps are Modify the default chain specification:

1. Generate the plain text chain specification: ./target/release/parachain-template-node build-spec --disable-default-bootnode > plain-parachain-chainspec.json
2. Modify the file to change para_id indicated by the user (flag --para_id)
3. Generate a raw chain specification file from the modified chain specification file: ./target/release/parachain-template-node build-spec --chain plain-parachain-chainspec.json --disable-default-bootnode --raw > raw-parachain-chainspec.json

Review Project Error Handling

Check error handling in all the code, for example the usage of expect. For example If something fails in the middle of an action shouldn't do a panic leaving files.

`pop build contract`: add default contracts directory

As a dev, I would like to have a seamless UX flow for building contracts in which I created with Pop, so that I don't have to waste time specifying configuration that should already by part of the Pop flow.

The UX flow for building contracts with Pop should be the following:

pop new contract my_contract

pop build contract my_contract

Currently it is not so, currently this is the flow:

pop new contract my_contract

pop build contract -p ./my_contract

-p should be an optional argument when the developer is storing their contracts in another directory other than the default.

Feature: Show the version of Polkadot when generating parachain

It may be useful to know what version of Polkadot to expect.

Test: `pop new parachain`

• Set up test environment
• The feature pop new parachain needs unit tests to cover all the functionality.

Pop Build - Parachain/Contract cargo project detection on current working dir

A discussion rather than an issue.

Currently, if pop build is invoked without a path, it builds any cargo project in the current working dir:

pub fn build_parachain(path: Option<PathBuf>) -> anyhow::Result<()> {
	cmd("cargo", vec!["build", "--release"])
		.dir(path.clone().unwrap_or("./".into()))
		.run()?;
	Ok(())
}

This is fine if the current working dir is a parachain project, but is unexpected or unintended behaviour when invoked in some other cargo project. pop build, for the sake of correctness, should be ideally scoped only to build contract and parachain cargo projects.
I understand that this is not really a problem, but this issue is meant for discussion on if it's worth the additional code or not.

docs: give credits to cargo contract

Like we have done for zombienet

Cargo test takes too long to test

Somewhat related to #81 and others, but cargo test is supposed to check and provide quick feedback for core features of pop for developers of pop rather than end-users. Since we are now building and checking a contract and a parachain when running cargo test it's extremely inefficient to check for coding errors when working on any feature.

Either feature gate or #[ignore] these long running tests so that they can be manually run by interested people, rather than having it run by default on cargo test.

Additionally, modify CI to include #[ignore]'d tests if so desired.

Also something worth considering is why are these tests not a part of the parachain/contract templates themselves?

Branch `add-pallet` needs to check for duplicated entries.

The command pop add pallet in branch add-pallet will inject the different features for the pallet independently of these being already present in the file.

This can be reproduce by running twice the same command, for instance: pop add pallet template -r runtime/src/lib.rs within a parachain project.

It will also instantiate multiple times the same pallet in the runtime.

Pop-CLI is big !

Currently, while developing and having RA on default-features (target) take up some 22.8 GB on my machine. We should explore ways to reduce build times, and possibly shave off heavy dependencies.

A fresh build from no target, takes up 6.9 GB.

Feedback: Check installation requirements

Feedback received:

Small nit in the documentation: cargo install doesn't work if you don't have nightly as default (or add +nightly)

It might be especially useful to get this checked via CI so we detect any regressions once resolved.

Feature: Contracts Verifiable Build

Due to the non-deterministic nature of Rust compilation, smart contract developers are advised to build their project inside a Docker container we provide.

Following the steps from use.ink documentation: https://use.ink/basics/verification/contract-verification/

1. Install Docker Engine
2. (Linux users) Make sure you complete the post-installation step. This is required for the correct operation of the command.
3. Ensure Docker Engine is up and running, and the socket is accessible.
4. Simply run cargo contract build --verifiable.

Feature: Contracts deployment

Add the functionality to enable deployment of smart contracts.

• Basic pop up contract feature: Deploy an ink! smart contract to a local development or remote parachain node.

feature: pop build contract debug

I had to build my contract in debug mode however there is no way to specify that

Removing `polkadot-launch` folder

I found the polkadot-launch folder in my project created from the base template. Do we want to include this (non-rust) when we are planning to offer "horde"?

`pop build parachain` fails within pop-cli repo folder

git clone https://github.com/r0gue-io/pop-cli
cd pop-cli
cargo build --release

./target/release/pop new parachain my-chain

./target/release/pop build parachain

This last command will not build the parachain as expected, instead it will output:

┌   Pop CLI : Building a parachain
│
    Finished release [optimized] target(s) in 0.89s
└  Build Completed Successfully!

Because it is looking in the current directory for ./target folder which is pop-cli's target folder.

Test: `pop new pallet`

The feature pop new pallet needs unit tests to cover all the functionality.

Fix add pallet template test

1bb0385

This test expects the test runtime test_lib.rs to be embedded in a default parachain project structure (The TomlEditor looks for the parent of dir runtime and expects it to be initialized with git before proceeding to apply any changes).

This test setup needs to be fixed:

1. git initialized workspace root
2. runtime and node folders
3. As of now, the test_lib.rs file is expected to be under $(workspace_root)/runtime/src/

Clean up of parachain template after `pop new parachain`

Feature: Contracts parachain template

With the feature pop new parachain add the functionality to generate a Smart Contracts parachain: A parachain that supports Wasm-based contracts such as ink!. This will be through integrating pallet-contracts with the Base Template.

Feature: `pop build parachain`

Add the command build parachain to build a parachain node:

pop build parachain <options>

For first version name the path of the repository where the parachain node is located.

Advanced features:

• Build a parachain, with new options: build the node (default), a native or deterministic runtime; or export the runtime wasm blob, genesis state, and chain spec: #28 and #29
• Cache #9

`pop build parachain`: validate endowment

It currently just takes a string, so user could enter anything which could break the build.

feat: adding cli functionality for zombienet

Some ideas that came to me today:

pop new zombienet: create a new zombienet toml file with relay
pop add zombienet --chain assethub --path zombienet.toml: add asset hub to zombienet toml file

Reason for it is that smart contract developers might want this to write integration tests for their smart contracts that use pallets and xcm functionality with other chains. By adding such features they don't have to learn zombienet.

Test: Mock build for unit testing

Test the pop build parachain, pop build contract and pop test contract is extremely slow because has to build the contract each time.

For unit testing we have to mock this behaviour.

`pop new parachain` should initialize a git repo

clone_and_degit is used to instantiate a new parachain. But having a first commit or even git init run is desirable.

CI: Testing generated parachains in `base-parachain`.

E2E tests involving cargo check | building a parachain (and to a lesser extent, an ink contract), can take a while on the CI runner. This slows us down for most tasks (pull requests) that do not pertain to modifying any of the current local *.templ template files which are used to customize the generated parachain.

One approach is to simply #[ignore] these tests, and have them run only on merge to main which is fine (#90). But here we discuss an alternative strategy:

pop new parachain is a result of sourcing base-parachain and applying the customization in the form of templ files that reside locally in this repo. The validation an e2e test like build_parachain makes is to effectively run pop new parachain and then run cargo build --release on it to make sure the generation was successful.

This checks two things:

1. The *.templ files are valid and up-to-date with the source template, in this case, base-parachain
2. The base-parachain builds without errors

This can be delegated to base-parachain itself, where one CI check on the base-parachain repository could download the latest pop release binary, run pop new parachain, which in turn uses the release templ files, and effectively check both the validity of the template files in pop-cli and the base-parachain itself.

`pop build parachain`: rename `parachain-template-node` binary to project name

Based on this PR comment: #30 (review)

`pop build parachain`: Export WASM and Genesis files

Add an option for pop build parachain to export WASM and genesis files (has to generate a chain spec file) after build it.

The steps are Prepare the parachain collator:

1. Export the WebAssembly runtime for the parachain: ./target/release/parachain-template-node export-genesis-wasm --chain raw-parachain-chainspec.json para-2000-wasm
2. Generate a parachain genesis state: ./target/release/parachain-template-node export-genesis-state --chain raw-parachain-chainspec.json para-2000-genesis-state

Feature: `pop add pallet`

Command pop add pallet that adds an existing FRAME pallet into a parachain runtime, along with a default configuration.

Options will allow specification based on pallet name or URI.

More options for `pop up contract`

Currently pop up contract does Upload + Instantiate at the same time.

Discuss if is worth it to have separate it in 2 steps like cargo-contracts: https://github.com/paritytech/cargo-contract/blob/master/crates/cargo-contract/src/main.rs#L126

Consider adding more options for deployment e.g cargo-contracts options: https://github.com/paritytech/cargo-contract/blob/master/crates/cargo-contract/src/cmd/mod.rs#L94

Feature: Contracts functionality

Add the functionality to enable seamless creation of smart contracts.

For that:

• Wrap cargo-contracts tool
• Basic pop new contract feature: Create a new ink! smart contract from a template
• Basic pop build contract feature: Build an ink! smart contract
• Basic pop test contract feature: Test an ink! smart contract by executing all unit tests.

Docs: Give credit to the tools being used by pop cli

error: could not find `Cargo.toml` in `/Users/bruno/Library/Caches/pop/.src/polkadot-sdk` or any parent directory

When I run the following command:

pop up parachain -f ./test_parachain/network.toml

I get the following error:

┌   Pop CLI : Deploy a parachain
│
▲  The following missing binaries are required: polkadot-v1.9.0
│  
◇  Would you like to source them automatically now?
│  Yes 
│
⚙  They will be cached at /Users/bruno/Library/Caches/pop
│  
◐  Sourcing polkadot-v1.9.0...                                                                                                               
error: could not find `Cargo.toml` in `/Users/bruno/Library/Caches/pop/.src/polkadot-sdk` or any parent directory
Error: command ["cargo", "build", "--release", "-p", "polkadot"] exited with code 101

`pop up contract`: run substrate-contracts-node if it does not exist

Based on this PR comment: #27 (comment)

For pop up contract, we should do a check, if there is no substrate node running on localhost:9944 then have Pop CLI run one.

Cache to improve UX

Internally to cache the path where the node was generated after execute pop new parachain. Then, on build, if the cache is non empty, we use that path.

Useful for pop build parachain #2 to don't have to specify the path when build the parachain recently generated.

And for pop add pallet #7 because for the runtime manipulation which will be part of add-pallet, either the runtime needs to be automatically inferred (depending on if pop new was called), or it needs to take a path arg.

Pop add pallet fails to correctly resolve relative paths when invoked inside the parachain repo.

Say you create a new parachain : pop new p my-chain
Ignoring the git status for now (maybe fixed by #65 - you can meanwhile cd my-chain, git add ., git commit), if you run pop add pallet from outside my-chain, specifying runtime path as -r my-chain/runtime/src/lib.rs, the cmd works.

However, if you were to cd into my-chain and provide the runtime path as -r runtime/src/lib.rs, the cmd fails:

❯ pop add p template -r runtime/src/lib.rs
T   Pop CLI : Adding pallet "pallet-parachain-template"!
|
Failed to open repository: failed to resolve path '': No such file or directory; class=Os (2); code=NotFound (-3)
Error: Workspace has uncommitted changes, aborting pallet addition

UX improvement for `pop add pallet` + `pop new pallet`

Problem statement:

From a UX perspective, it is perhaps, quite natural for someone to expect that pop add pallet template pertains to adding a ready-to-hack pallet into their workspace, without having to divvy up the task between pop new pallet and a subsequent pop add pallet -p <path to new pallet> just to get a new pallet working seamlessly. [note: the -p or path to the pallet crate isn't implemented in code yet]. new pertains to creation of new objects such as contracts, parachains, zombienet config files, etc. but add should aim to be like add this thing to my stuff but don't break anything while doing so, which is exactly the opposite of what pop add pallet template currently does. It requires pop new pallet to be run before & that too in a pallets folder because that's the path searched by pop add pallet template without any args such as the aforementioned and unimplemented -p, --path [pallet-path]

I've bought up this design decision before, that at least pop add pallet should internally take care of placing a ready to go pallet-parachain-template in a standard path $(workspace_root)/pallets/ and edit the dependency manifest and runtime file to instantaneously ready pallet development. This ease and swiftness is what I believe to be the core pillar of pop & should be the guiding principle when designing interactions.

Solution

pop new pallet remains, but the logic of pop add pallet is altered such that it already makes a pallets/ dir and runs pop new pallet internally so that after running pop add pallet template the user can immediately cargo build/check their code without having to see the dreaded missing dependency errors.

Related PRs #62, #43

`pop up parachain` can't launch local network

Steps:

• Build pop from current master
• Run pop up parachain --file ./network/rococo.toml at pop-node folder.
  rococo file from -> https://github.com/r0gue-io/pop-node/blob/main/networks/rococo.toml
  Observed behaviour:

┌   Pop CLI : Deploy a parachain
│
⚙  Launching local network...
│  
2024-03-06T15:11:36.665016Z  INFO pop::parachains::zombienet: spawning network...    
2024-03-06T15:11:36.682006Z  INFO zombienet_orchestrator: 🧰 ns: zombie-e6e17946-6fb7-47ed-9696-386b98ffb96a
2024-03-06T15:11:36.682017Z  INFO zombienet_orchestrator: 🧰 base_dir: "/tmp/zombie-e6e17946-6fb7-47ed-9696-386b98ffb96a"
└  Could not spawn network: Generator error

Generator error comes from upstream, zombienet-sdk.

Feature: Pallet Removal

As suggested at sub0, it would be great to be able to remove a pallet from a runtime.

Add `CONTRACTS_NODE_PATH` to Pop when running e2e tests for contracts

Based on this PR comment: #22 (comment)

The idea would be devs can run e2e tests like this:

pop test contract  -p ./my_contract --features e2e-tests -n /Users/bruno/src/substrate-contracts-node/target/debug/substrate-contracts-node

`pop new parachain`: remove pallet template from base template

When executing pop new parachain we don't want there to be a

• template folder in pallets
• remove pallet_parachain_template from runtime/src/lib.rs
• remove " " from Cargo.toml
• remove " " from runtime/Cargo.toml

`pop up contract`: build contract if it has not been built

Based on this PR comment: #27 (comment)

For pop up contract, we should do a check, if the contract has not been built, then build it.

`pop new parachain`: add zombienet config

Add a default zombienet config file when generating a new parachain from a template.

Feature: `pop up parachain`

Add new command pop up parachain that spawn a local network using Zombienet and a configuration file.Deployment providers will be added in the future to support additional deployment targets.

Change `vanilla` template naming to `base`

This has been decided internally and should be changed before sub0