nomad-xyz / rust Goto Github PK

• want to set per-network gas limits for home.update, replica.update, replica.prove, replica.process
• want to set per-network gas limits and prices for all fraud-related methods (double_update, improper_update, owner_unenroll_replica, unenroll_replica)
• Can get gas limits for non-fraud methods from past data
• Will need to use a gas reporter tool for fraud-related methods

• Describe what the agent does, namely forward signed updates from the home to configured replicas
• Describe how configuration maps to relayer behavior (forwards updates for channels home to X replicas)
• Describe behavior during failure modes for clarity

Current agent configuration scheme is rigid and breaks when trying to supply external config file. Want to allow any arbitrary party to provide a unified config file and set of env vars to run agent against arbitrary networks.

• Fix base config validation to allow for config files
• Compare secrets against supplied config (whether builtin or from file)

Want to avoid subsidizing expensive txs on ethereum. Safer this way.

Missing definitions for

• Agent-specific config types
• SignerConf
• ChainConf
• Logging types

copied from a Discord message which should be transcribed:

the code for deploying the contracts is here: https://github.com/nomad-xyz/monorepo/tree/main/packages/new-deploy

this script is an example of invoking the deploy script: https://github.com/nomad-xyz/monorepo/blob/5b603ee6f58c27d9d5dda6b2a1fa9435a9fc9b97/packages/new-deploy/package.json#L31
the arguments are

1. the local path to the config file
2. the local path to the overrides for submitting transactions (if desired)

in my case, i opted to store the configs in a local file in the same package, here: https://github.com/nomad-xyz/monorepo/tree/main/packages/new-deploy/config/development

the layout of the config file should look like this: https://github.com/nomad-xyz/rust/blob/main/configuration/configs/development.json

steps to make your own deploy are:

1. configure the chains you wan to deploy in the protocol section
2. delete the core and bridge components so that they are empty objects {} (TODO: make an example pre-deploy configuration)
3. plop that file into the local path, like above, and invoke the deploy script - just like in npm run deploy

When CamelCased config variables show up in the agent config, the default behavior by the config crate is to treat it as a word boundary, resulting in weird behavior in the environment variable pipeline.

@prestwich would this be fixed in your config update?

Document steps for running a Nomad agent in general. This would include some of the following:

• Where/how to get/setup image for agent (cc @yourbuddyconner)
• Supplying configuration file or specifying run env
• Supplying env vars for RPC endpoints, transaction signers, and optional attestation signer

Additional Relevant Deploy Info:

• We release images via Github actions to our GCR Image Repository which get tagged with the git sha/branch they were built with
• You can see the build process here if you'd like to build from source -- here is the Dockerfile.
• We've got a helm repository here, with a nomad-agent helm chart.

• spin up two providers
• compare results
• dispatch a tx
• compare results

color eyre is cute, put a little messier to read. we can always add a colored subscriber if we want

• Some reads want timelag (updater produce_update), some don't (processor message_status)
• Having all or nothing leads to unexpected behavior with agents reading old state
• Want granular control over when timelag is used per contract call at call time
• Want something like home.produce_update(TimeLag::On)

Duplicated code between tests for per-agent configuration. Would be nice to consolidate somehow with macro (maybe some test hook that does all the shared test ops).

Like the one in the monorepo :)

• want to reduce number of env vars passed
• signers and rpcs (connections) require "types" (http vs ws, hexkey vs aws)
• simplified deser. scheme without explicit types would simplify operational load

This function does not exist but is imported and used

Currently we dispatch transactions at various points in the codebase, however we don't explicitly check to ensure outcome of the transaction was successful. This results in loops and bad behavior.

Ex: https://github.com/nomad-xyz/agents/blob/main/agents/updater/src/submit.rs#L57

Solution:
Lift the dispatch code up into nomad-core and fix the problem, ensuring that all agents receive this change.

https://doc.rust-lang.org/std/string/struct.String.html#method.strip_prefix

this was standardized sometime after we wrote the strip_0x_prefix method

• IntGaugeVec tracking number of channel faults (channel_faults)
• channel_faults split up into IntGauges by keys ["home", "replica", "agent"]
• Graph: show number of channel faults per agent per home <> replica channel
• Alerts: if any channel-specific IntGauge for channel faults exceeds say 10 faults in 1 hour, send alert in slack

TL;DR the bug in the updater that failed all dev homes is as follows:

• Updater tries produce new_roots from a local merkle tree which is built from all the emitted messages for that home (same process as the processor)
• The updater has until now never synced/indexed messages
• The updater started syncing messages but before that has finished, it picks the current root of the local tree and uses that as its new_root
• This update of last_committed-->new_root is invalid (the new_root is actually a really old root)
• The improper update is submitted and the home is failed (for each updater)

Solution:

• Have the syncing merkle tree keep track of the current committed_root corresponding to its current root
• I.e. Each time the syncing tree stores a new leaf, it will get the Dispatch.committed_root field and update its current committed_root field
• The updater will only use tree.root() if tree.current_committed() == old_root

Import ethers rather than importing sub crates of ethers individually.

#30 (comment)

High Level Agent Startup Flow

1. Retrieve NomadConfig for the intended environment (dev, staging, prod)
2. Use NomadConfig to build union of nomad_base::Settings and agent-specific settings
3. Instantiate agent from settings (e.g. Updater::from_settings(settings).await? in agents/updater/main.rs) and run agent (implementation already there)

Implementation Details (modify decl_settings! macro)

• Implement nomad_base::Settings::from_nomad_config(config: NomadConfig, home_name: &str) to map NomadConfig to agent-specific settings (used in next step)
• fn new() -> Result<Self> must:
  • Fetch NomadConfig based on the RUN_ENV env variable by calling config::get_builtins(<environment name>)
  • Get separation of home vs. replicas based on the AGENT_HOME env variable
  • Return result of Settings::from_nomad_config(config, home_name)
• Agent takes result of Settings::new()? and uses to run agent (rest of agent/main.rs)

Loading secrets (signers & RPC)?

• Looks like new config crate is not meant to support secrets since all NomadConfig values will come directly from public crate imported as dependency (using something like config::get_builtins(...))
• Possible solution:
  • Create new AgentSecrets struct (below) that can be deserialized from JSON (shown below)
  • In agent-specific Settings::new() function, expect/load a secrets.json file (below) that deserializes into AgentSecrets (inject secrets into json file as we already do)
  • Implement some kind of Settings::from_configuration(config: NomadConfig, secrets: AgentSecrets, home_name: &str) where all public values come from NomadConfig while all private values come from AgentSecrets
  • Implement checks if we are missing values in AgentSecrets when comparing to NomadConfig (e.g. we accidentally left out rpc for one network)

=== Rust - AgentSecrets ===
struct AgentSecrets {
   rpcs: HashMap<String, ChainConf>,
   transaction_signers: HashMap<String, SignerConf>,
   attestation_signer: SignerConf,
}

=== JSON - secrets.json ===
{
   "rpcs": {
       "ethereum": {
           "type": "http",
           "url": ""
       },
       "moonbeam": {
           "type": "http",
           "url": ""
       },
       "milkomedaC1": {
           "type": "http",
           "url": ""
       },
   },
   "transactionSigners": {
      "ethereum": {
          "key": "",
          "type": "hexKey"
      },
      "moonbeam": {
          "key": "",
          "type": "hexKey"
      },
      "milkomedaC1": {
          "key": "",
          "type": "hexKey"
      },
   },
   "attestationSigner": {
       "key": "",
       "type": "hexKey"
   }
}

• secrets and base settings validation fail because they match against the configuration crate
• we want to allow outside parties to provide their own configs that might not match ours and don't want to crash agent because of it

• rollback ethers version to remove last updater patch (adds timelag for contract state reads)
• produce updates from local tree of messages which uses indexing timelag

currently reverts in certain actions are not caught in pre-flight because we no longer estimate gas. We should run an eth_call to see if they would cause a revert and take appropriate action

example action:
update on a failed home

#30 (comment)

Can build AgentSecrets from a secrets.json file. Should also add method to load secrets directly from environment variables.

Should have something like BaseAgentSettings in configuration and then all the types with lots of functionality (e.g. CachingHome) implement From<BaseAgentSettings>. Overall, we should move from implementing <config struct>::into_<some struct with functionality> to implementing <struct with functionality>::from_<some config struct>

Not used anywhere. Collateral from earlier fall 2021 days. Delete.

Example file in nomad-base can also go.

• backtracking algorithm for catching updates only works if we assume there will be more updates to reveal a discontinuity in RPC
• relayer and watcher (which sync at tip) are then vulnerable to missing updates completely for hours until next update shows up (if it ever does)
• we want to index at the tip and also reindex ranges finality_blocks behind the tip to catch any missed updates

Want to allow agent-specific values to be overridden through environment variables.

• implement EnvOverridable for each agent config in configuration/src/agent
• agent_settings.load_env_overrides() in nomad-base/src/settings/macro.rs before adding to overarching agent settings

#35 (comment)

• Will follow same pattern as Channel struct in run refactor PR
• Each agent has a type AgentMetrics and must implement a method build_agent_metrics which specifies how to build the struct from AgentCore

See comment here for details on executing the secrets_template script.

We need to write an automation tool that wraps the secrets_template script, and diffs the output of it with a secret that exists in Google Secrets Manager and outputs a patch to the existing secret.

This tool can then be used as a pre-deploy hook to error if the secrets are malformed or missing entries.

• Describe the functionality of the processor, namely that it tracks messages on the home and proves/processes them on any configured replicas
• Describe how configuration maps to processor behavior (proves/processes messages for channels home to X replicas
• Describe the functionality of the allowlist, denylist, index_only, and s3 options and how they can be used for custom clients/use cases
• Describe behavior during failure modes for extra clarity

Suite of tasks for UX around allowing external parties/stakeholders to run agents

Problem

• Currently, updater must wait till its last update becomes final to move onto producing a new one (does so by re-reading its own update on a timelag)
• This incurs a blocktime x finality latency for between each update
• If you dispatch a message that reaches finality just after the last update was submitted, you have a potentially 2 x (blocktime x finality) latency

Solution

• We know that the all updates produced in db will be valid chain, thus the new_root of the prev produced update will always be the old_root of the next produced update
• UpdateProducer should immediately retrieve next root to build off of from produced updates db

We need to be able to call the secrets_template binary from within the docker container for the purposes of automation tooling. Please add it to the docker build for the agents.

• Describe key provision process and watcher enrollment on xapp connection managers
• Describe what watcher configuration with given homes and replicas means (watchers over channel from home to X replicas)
• Suggest running watcher with multiple RPC endpoints for safety
• Describe behavior during failure modes

Need some way to override agent config values at agent run time.

Nomad Team:

1. Run pre-agent script, which takes in a RUN_ENV environment variable. This script will output a template secrets.json and the a config.json from the Nomad configuration crate for that RUN_ENV (config will be development or production json files in crate).
2. Override any necessary values in secrets.json or config.json.
3. Run agent with RUN_ENV and AGENT_HOME. All agent expects is secrets.json and config.json in cwd.

External Party (no usage of Nomad configuration crate)

1. Create custom config.json and secrets.json in cwd (might not include all our networks).
2. Run agent.

Just watcher:

• Gauge: number of updates sent through UpdateHandler (number of updates checked)
• Gauge: 0 or 1 binary for double update detected
• Histogram: latency between Update event emitted and update sent through UpdateHandler (see ContractSync for example)

All agents that check home failure:

• Gauge: 0 or 1 for improper update observed (failed home; can be for all agents)
• Gauge: number of home failed checks (can be for all agents)

• Add grafana display for the number of double updates a watcher has observed (fine if 0, go crazy with alerts if >= 1)

• Add grafana graph for number of updates the watcher has inspected for double updates (should be up and to the right if watcher is functioning properly)

• Add grafana display for number of times home has been observed as failed (fine if 0, go crazy with alerts if >= 1)

• Add grafana graph for number of home failure checks an agent has performed (should be up and to the right if agent is healthy)

• updater and watcher cannot use the run_many pattern and thus won't catch errors for channel faults
• at some point, need to rearchitect to allow them to catch errors instead just crashing