paritytech / metadata-portal

Metadata portal for Parity Signer

Metadata Portal 🌗

Metadata Portal is a self-hosted web page that shows you the latest metadata for a given network.

This is an important addition to Signer, which can update the metadata inside only through a special video QR code without going online. Parity will host its own version of the page for all chains for which we sign the metadata. External users (chain owners) will be able to deploy their versions of metadata portal if they want.

How does it work?

It all starts with the Github repository. Any user can clone it and run their Metadata Portal. We also host our own version, so let's break down the principles of working on it.

Metadata Portal supports two metadata sources in parallel. Both are equally important for different types of users.

1. Parsing it from chain and generating QR codes itself

This flow is important for all users who want to always have the latest metadata in their signing devices to parse and sign their transactions right away.

• Cron job
  • runs every N hours and checks every known network for the latest metadata version
  • If any network has a new version of metadata that has not yet been published on the Metadata Portal
    • generates unsigned metadata QR code
    • creates new pull request to the repo
    • sends notification to a Matrix channel
• Release manager
  • checkouts pull request's branch locally
  • runs make signer locally to sign new metadata using his signing air-gapped device
  • commit and push changes to the same branch
• Owner of the repository
  • accept and merge the PR
• Github action is triggered to regenerate and re-deploy the Github Page

2. Showing manually uploaded and signed QR codes via PRs

This flow is for security-oriented users and Parity itself. It allows chain owners to sign their metadata updates and host QR codes for their users.

• Release manager generates a new signed QR code manually in an air-gapped environment using his signing device
• He opens a PR and signs commits by his YubiKey to prove its validity
• Owner of the repository accepts the PR
• Github action is triggered to regenerate and re-deploy the Github Page

Deployment

Requirements

1. install https://github.com/paritytech/parity-signer to your signing device

Steps

You can use Github Pages to host the metadata-portal for your set of chains

1. Fork this repo
2. Edit config.toml
   1. Add/remove chains
   2. Edit signer's name and public key. The key can be exported from parity-signer
3. Configure GitHub Pages to build from gh-pages branch (Settings -> Pages -> Source)
4. Edit domain name in:
   1. homepage field in package.json
   2. public/CNAME file
5. Notifications to Matrix:
   1. You can disable it by setting NOTIFY_MATRIX: false in .github/workflows/update.yml
   2. Otherwise, add MATRIX_SERVER, MATRIX_ROOM_ID, MATRIX_ACCESS_TOKEN values to project Actions secrets

Development

Dependencies

The main requirement is the OpenCV. You can check this manual: https://crates.io/crates/opencv

Arch Linux:

OpenCV package in Arch is suitable for this.

pacman -S clang qt5-base opencv

Ubuntu:

sudo apt install libopencv-dev clang libclang-dev

Other Linux:

You have several options of getting the OpenCV library:

• install it from the repository, make sure to install -dev packages because they contain headers necessary for the crate build (also check that your package contains pkg_config or cmake files).

• build OpenCV manually and set up the following environment variables prior to building the project with opencv crate:

  • PKG_CONFIG_PATH for the location of *.pc files or OpenCV_DIR for the location of *.cmake files
  • LD_LIBRARY_PATH for where to look for the installed *.so files during runtime

Additionally, please make sure to install clang package or its derivative that contains libclang.so and clang binary.

• Gentoo, Fedora: clang
• Debian, Ubuntu: clang and libclang-dev

MacOs:

brew install opencv

If you're getting dyld: Library not loaded: @rpath/libclang.dylib: OS can't find libclang.dylib dynamic library because it resides in a non-standard path, set up the DYLD_FALLBACK_LIBRARY_PATH environment variable to point to the path where libclang.dylib can be found, e.g. for XCode:

export DYLD_FALLBACK_LIBRARY_PATH="$(xcode-select --print-path)/Toolchains/XcodeDefault.xctoolchain/usr/lib/"

Frontend

Before running the frontend locally, you need to generate a data file:

make collector

And then run the app in the development mode

yarn start

metadata-portal's Issues

Add Rococo Metadata

It would be really useful to have Rococo supported on the Metadata portal.

KUSAMA Meta Data update

The Metadata on https://metadata.parity.io/?tab=1#/kusama is out of data which is preventing to do an important and urgent Governance action

Metadata outdated for polkadot/9431

The metadata portal does not contain the latest metadata for Polkadot/9431 and states

The metadata for polkadot Network is out of date. Request the new metadata version by creating a Github issue.

Thus, hereby I am doing that.

Feat req.: Add ability to download data embeded in QR codes

The portal's various QR codes don't include any way to get the data that is present in the QR codes. The metadata portal should provide that as a raw download link too, not just the QR.

Related to #28 but also just for convenience, this option would be a user friendly way to get that data for this portal as well as https://github.com/novasamatech/metadata-portal fork of this repo, where various parachains are listed.

Add parachains with the space character in the metadata name

There is a problem with signing QR codes for parachains with the space character in the name.
For example crab parachain contains the space in the metadata.
When the metadata and chains specs are going to be signed the error occurs.

cargo run --release sign
   Compiling metadata-cli v0.1.0 (/Users/pgolovkin/work/omni/metadata-portal/cli)
    Finished release [optimized] target(s) in 7.59s
     Running `target/release/metadata-cli sign`
✔ Choose file to sign · unsigned_crab parachain_specs.png
[2022-07-10T21:54:58Z ERROR] error showing QR code: Usage: /opt/homebrew/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/webbrowser.py [-n | -t] url
        -n: open new window
        -t: open new tab
    
make: *** [signer] Error 1
pgolovkin@Pavels-MacBook-Pro metadata-portal % 

Please fix this problem or share some thoughts how it's better to fix the problem: the show QR process fix or generate QR process fix.

Save metadata source

When metadata is fetched from network, record block it was generated for. When it is generated from github, identify source as well (wasm hash maybe?)

This is needed to track silent metadata changes. One just happened recently in Acala. This severely breaks signer functionality (other clients functionality as well probably) and might result in harmful consequences, so we should actively fight it.

How add other network like Efinity?

how can I add network in metadata parity https://metadata.parity.io/ there are only 3 network available or is there any other site with complete network?

Redesign

Redesign the metadata-portal according to the mockup
https://www.figma.com/file/eOnz3xbzObTG8hQlIRs6jg/Metadata-Portal?node-id=0%3A1

URL can show the selected network

it's important to have selected chain appear in the URL, so people can share links pointing to the particular chain. For example https://metadata.parity.io/#/kusama would automatically select kusama network.

Polkavault update metadata trouble

Hi, I have been using polkavault for a while to stake and unstake DOT and TAO.

Today, when signing a transaction, the app asked me to update the metadata onnode-subtensor Network Metadata.

All the metadata is on the latest version, i scanned polkadot and subtensors data, but i keep getting the same message.

I dont know what i am doing wrong.

Hope someone could advice. Thanks

Portal doesn't display additional chains properly

I've forked the repo and added statemine and statemint. However, they don't get populated with the correct metadata and info. When you select either of the new chains, the portal instead shows the info of the last chain selected (among Polkadot, Kusama and Westend). You can see the issue here:
https://michalisfr.github.io/metadata-portal/

Testing locally with adding and removing chains, I noticed that if I remove some of the main chains (from config.toml) but keep statemine and/or statemint, the page completely fails to render and displays an empty page. If I reorder the chains, the page renders but the issue remains.

It's worth noting that chains.json is populated with the correct information after running make collector.

Add api to fetch signed metadata updates data prior to erasure frames generation

Just give some api so that tools could fetched signed network specs and metadata (same content that's erasure-encoded in QR video) directly for client-side encoding.

This is needed in particular for tools like Kampela where the payload should be encoded into device-signed (on top of authority certificate signature) NFC fountain, which could only be done client-side. Splitting into packets is done client-side as well, as packet size could be variable. Only signature ensuring untampered original update content is really needed here, but serving content with the signature seems trivial here at this point and would be quite convenient.

Empty page when metadata-portal is loaded for the 1st time

Empty page when metadata-portal is loaded for the 1st time

The problem is with App.tsx

 const [currentNetwork, setCurrentNetwork] = useState<string>(
    (Object.keys(allChains).includes(location) && location) ||
      (localStorageNetwork && localStorageNetwork.toLowerCase()) ||
      Object.keys(allChains)[0]
  );

The localStorage is empty and allChains are not loaded.

Phone cannot scan Metadata QR codes.

I'm using a Pixel 4a (5g) with Graphene Bramble 2023032600. Whenever I try to scan the Polkadot metadata (any metadata) it never scans. On occasion, I receive the prompt about holding device still, and it never moves past 2/x. I've tried cleaning the camera lens with no luck. Other dot QR codes scan fine, including network specs. Is there any means of improving readability of the QR code?

Add github link to the site

if we want people to start copying it

Polkadot Metadata Update not available - Polkadot Vault

Greetings,

The polkadot metadata fountain has dried up. The site states that it needs and update and to create a Github issue. Does anyone know if there is a release pending?

Thanks,
8.0.

Use github repo for different chains

The problem

The metadata portal with different parachains support is hosted. I'd like to user the update from github repositories approach for parachains where it's possible. However I can't find a way how the github repo may be specified for each parachain. For example for Acala and for Statemine.

Could you please point me how it may be achieved or work on the solution for it?

Polkadot Vault incorrect website statement

New to the app and connecting to networks. For relay chains it says to go to parity.metadata.io however it is metadata.parity.io. Don't know how to bring this issue to Polkadot Vault maintainers directly so I hope this will work

Decrease playback speed of metadata QR code as a solution for older/basic phones

Explained here: novasamatech/parity-signer#2048

Using an older or more basic smartphone, the metadata QR code has such a high playback speed that the camera cannot track the QR code between frame updates. It is very obvious this is the issue because the QR code is quite blurry despite being in focus.
Increasing the display size of the metadata QR code did not remedy this problem, as the camera simply cannot keep up with such a quickly changing image.

Using this tool: https://ezgif.com/split I was able to save the animated metadata QR code manually and reduce the playback speed of it, and this helped significantly to be able to scan the code fully.

It would be great if the Metadata Portal could have a reduced speed of playback for the animated metadata QR code for older or more basic smartphones.

Adding new network logos to Parity Signer

We are working with a Parachain that wants to add their network logo to Parity Signer who would really appreciate being able to have this feature.

Oops

Error while starting locally

When trying to run project locally I get the following error:

Compiled with problems:X

ERROR in ./src/scheme.ts 3:0-37

Module not found: Error: Can't resolve './chains.json' in '/home/wirednkod/Documents/repos/parity/metadata-portal/src'


ERROR in src/scheme.ts:1:22

TS2307: Cannot find module './chains.json' or its corresponding type declarations.
  > 1 | import jsonData from "./chains.json";
      |                      ^^^^^^^^^^^^^^^
    2 |
    3 | export interface ChainSpec {
    4 |   name: string;


ERROR in src/scheme.ts:26:35

TS7006: Parameter 'chain' implicitly has an 'any' type.
    24 |
    25 | export function getChains(): Chains {
  > 26 |   const chainList = jsonData.map((chain) =>
       |                                   ^^^^^
    27 |     Object.assign({} as ChainSpec, chain)
    28 |   );
    29 |   return chainList.reduce((obj, chain) => {


ERROR in src/scheme.ts:29:28

TS7006: Parameter 'obj' implicitly has an 'any' type.
    27 |     Object.assign({} as ChainSpec, chain)
    28 |   );
  > 29 |   return chainList.reduce((obj, chain) => {
       |                            ^^^
    30 |     return {
    31 |       ...obj,
    32 |       [chain.name]: chain,


ERROR in src/scheme.ts:29:33

TS7006: Parameter 'chain' implicitly has an 'any' type.
    27 |     Object.assign({} as ChainSpec, chain)
    28 |   );
  > 29 |   return chainList.reduce((obj, chain) => {
       |                                 ^^^^^
    30 |     return {
    31 |       ...obj,
    32 |       [chain.name]: chain,
    ```

Support networks with metadata V12

Is there any way to generate the metadata for the networks with metadata v12 (like Edgeware)?

Metadata for Polkadot Network is out of date

The metadata fountain welcomes the user with the message:

The metadata for polkadot Network is out of date. Request the new metadata version by creating a Github issue.

https://metadata.parity.io/?tab=1#/polkadot

Polkadot metadata

Hi!
The metadata for polkadot Network is out of date.
Can you make the latest available?
Thx!

Remember selected network

As a user,
When I choose a network on the selector
I'd like it to be preserved between page refreshes.
So I don't have to select it every time manually.

"Add to signer" button is confusing

novasamatech/parity-signer#1031 (comment)

This is not the first time I see users confused about that button. We need to make in more clear apparently.

update Encointer network spec and metadata

redirected from:
https://metadata.parity.io/?tab=1#/kusama-encointer-parachain

We have an upcoming use case where vault support would be great.

The metadata for polkadot Network is out of date.

The metadata for polkadot Network is out of date. Request the new metadata version by creating a Github issue.
Create a Github issue

Add mechanism to verify metadata independently.

Now that we have metadata published and other developers can deploy forks and mirrors of this portal, we should start building usable infrastructure for verifying metadata validity. We currently use some development tools for this on metadata updates, but we should make metadata verification easily accessible for everyone even if they are not subscribed to our verifier certificate.

Thus, we should:
[ ] Add metadata hash to update screen
[ ] Add metadata hash identicon to update screen (it's just normal polkadot identicon build on metadata hash - it's 2 hashed in a row, sacrifice elegance for modularity)
[ ] Make some tool(s?) users can use to query metadata hash through rpc calls locally or from web
[ ] Add same information (and tools?) to popular places like polkadot-js website

Support a list of `rpc_endpoint` for collecting the metadata

Support a list of rpc_endpoint for collecting the metadata.

The problem
Some of RPC nodes are not available when updating the metadata for networks. In that case the update process fails.

The solution

1. Add a list of rpc_endpoint to the config.toml file.
2. During the update if the 1st RPC node is not available check 2nd, 3rd, etc. nodes.
3. If all RPC nodes are not available then fail.

Show the state when the chain was added on the metadata portal but it's metadata is outdated for some reason

Show the state when the chain was added on the metadata portal but it's metadata is outdated for some reason

The case:

1. Metadata portal updates the metadata for all added network into the chains.json file
2. After the update Metadata portal deplyes new QR codes
3. During the metadata deploy metadata portal realises that for some chain metadata is already outdated (while collect is working)
4. As a result network with outdated metadata disappear from the metadata portal that confuse users.

Invalid cross-device link when building updater.

I'm trying to build the Updater on an Arch System. Whenever I run make updater I receive error message of:

cargo run --release update
Finished release [optimized] target(s) in 0.40s
Running target/release/metadata-cli update
[2023-01-29T22:05:19Z INFO ] ⚙️ Generating unsigned_kusama_metadata_9370.apng...
[2023-01-29T22:05:22Z ERROR] Invalid cross-device link (os error 18)
make: *** [Makefile:14: updater] Error 1