nomad-xyz / rust Goto Github PK
View Code? Open in Web Editor NEWRust work for nomad actors
License: Apache License 2.0
Rust work for nomad actors
License: Apache License 2.0
Current agent configuration scheme is rigid and breaks when trying to supply external config file. Want to allow any arbitrary party to provide a unified config file and set of env vars to run agent against arbitrary networks.
Want to avoid subsidizing expensive txs on ethereum. Safer this way.
Missing definitions for
copied from a Discord message which should be transcribed:
the code for deploying the contracts is here: https://github.com/nomad-xyz/monorepo/tree/main/packages/new-deploy
this script is an example of invoking the deploy script: https://github.com/nomad-xyz/monorepo/blob/5b603ee6f58c27d9d5dda6b2a1fa9435a9fc9b97/packages/new-deploy/package.json#L31
the arguments are
in my case, i opted to store the configs in a local file in the same package, here: https://github.com/nomad-xyz/monorepo/tree/main/packages/new-deploy/config/development
the layout of the config file should look like this: https://github.com/nomad-xyz/rust/blob/main/configuration/configs/development.json
steps to make your own deploy are:
{}
(TODO: make an example pre-deploy configuration)npm run deploy
When CamelCased config variables show up in the agent config, the default behavior by the config crate is to treat it as a word boundary, resulting in weird behavior in the environment variable pipeline.
@prestwich would this be fixed in your config update?
Document steps for running a Nomad agent in general. This would include some of the following:
Additional Relevant Deploy Info:
color eyre is cute, put a little messier to read. we can always add a colored subscriber if we want
home.produce_update(TimeLag::On)
Duplicated code between tests for per-agent configuration. Would be nice to consolidate somehow with macro (maybe some test hook that does all the shared test ops).
Like the one in the monorepo :)
This function does not exist but is imported and used
rust/nomad-core/src/test_output.rs
Line 6 in 59282f6
Currently we dispatch transactions at various points in the codebase, however we don't explicitly check to ensure outcome of the transaction was successful. This results in loops and bad behavior.
Ex: https://github.com/nomad-xyz/agents/blob/main/agents/updater/src/submit.rs#L57
Solution:
Lift the dispatch code up into nomad-core
and fix the problem, ensuring that all agents receive this change.
https://doc.rust-lang.org/std/string/struct.String.html#method.strip_prefix
this was standardized sometime after we wrote the strip_0x_prefix
method
channel_faults
)channel_faults
split up into IntGauge
s by keys ["home", "replica", "agent"]
TL;DR the bug in the updater that failed all dev homes is as follows:
Solution:
committed_root
corresponding to its current root Dispatch.committed_root
field and update its current committed_root
fieldtree.root()
if tree.current_committed() == old_root
Import ethers
rather than importing sub crates of ethers individually.
High Level Agent Startup Flow
NomadConfig
for the intended environment (dev, staging, prod)NomadConfig
to build union of nomad_base::Settings
and agent-specific settingsUpdater::from_settings(settings).await?
in agents/updater/main.rs) and run agent (implementation already there)Implementation Details (modify decl_settings!
macro)
nomad_base::Settings::from_nomad_config(config: NomadConfig, home_name: &str)
to map NomadConfig
to agent-specific settings (used in next step)fn new() -> Result<Self>
must:
NomadConfig
based on the RUN_ENV
env variable by calling config::get_builtins(<environment name>)
AGENT_HOME
env variableSettings::from_nomad_config(config, home_name)
Settings::new()?
and uses to run agent (rest of agent/main.rs)Loading secrets (signers & RPC)?
NomadConfig
values will come directly from public crate imported as dependency (using something like config::get_builtins(...)
)AgentSecrets
struct (below) that can be deserialized from JSON (shown below)Settings::new()
function, expect/load a secrets.json
file (below) that deserializes into AgentSecrets
(inject secrets into json file as we already do)Settings::from_configuration(config: NomadConfig, secrets: AgentSecrets, home_name: &str)
where all public values come from NomadConfig
while all private values come from AgentSecrets
AgentSecrets
when comparing to NomadConfig
(e.g. we accidentally left out rpc for one network)=== Rust - AgentSecrets ===
struct AgentSecrets {
rpcs: HashMap<String, ChainConf>,
transaction_signers: HashMap<String, SignerConf>,
attestation_signer: SignerConf,
}
=== JSON - secrets.json ===
{
"rpcs": {
"ethereum": {
"type": "http",
"url": ""
},
"moonbeam": {
"type": "http",
"url": ""
},
"milkomedaC1": {
"type": "http",
"url": ""
},
},
"transactionSigners": {
"ethereum": {
"key": "",
"type": "hexKey"
},
"moonbeam": {
"key": "",
"type": "hexKey"
},
"milkomedaC1": {
"key": "",
"type": "hexKey"
},
},
"attestationSigner": {
"key": "",
"type": "hexKey"
}
}
currently reverts in certain actions are not caught in pre-flight because we no longer estimate gas. We should run an eth_call to see if they would cause a revert and take appropriate action
example action:
update on a failed home
Can build AgentSecrets from a secrets.json file. Should also add method to load secrets directly from environment variables.
Should have something like BaseAgentSettings
in configuration and then all the types with lots of functionality (e.g. CachingHome
) implement From<BaseAgentSettings>.
Overall, we should move from implementing <config struct>::into_<some struct with functionality>
to implementing <struct with functionality>::from_<some config struct>
Not used anywhere. Collateral from earlier fall 2021 days. Delete.
Example file in nomad-base can also go.
finality_blocks
behind the tip to catch any missed updatesWant to allow agent-specific values to be overridden through environment variables.
EnvOverridable
for each agent config in configuration/src/agent
agent_settings.load_env_overrides()
in nomad-base/src/settings/macro.rs before adding to overarching agent settingsChannel
struct in run refactor PRAgentMetrics
and must implement a method build_agent_metrics
which specifies how to build the struct from AgentCore
See comment here for details on executing the secrets_template script.
We need to write an automation tool that wraps the secrets_template
script, and diffs the output of it with a secret that exists in Google Secrets Manager
and outputs a patch to the existing secret.
This tool can then be used as a pre-deploy hook to error if the secrets are malformed or missing entries.
allowlist
, denylist
, index_only
, and s3
options and how they can be used for custom clients/use casesSuite of tasks for UX around allowing external parties/stakeholders to run agents
Problem
blocktime x finality
latency for between each update2 x (blocktime x finality)
latencySolution
new_root
of the prev produced update will always be the old_root
of the next produced updateUpdateProducer
should immediately retrieve next root to build off of from produced updates dbWe need to be able to call the secrets_template binary from within the docker container for the purposes of automation tooling. Please add it to the docker build for the agents.
Need some way to override agent config values at agent run time.
Nomad Team:
pre-agent
script, which takes in a RUN_ENV
environment variable. This script will output a template secrets.json
and the a config.json
from the Nomad configuration crate for that RUN_ENV
(config will be development or production json files in crate).secrets.json
or config.json
.RUN_ENV
and AGENT_HOME
. All agent expects is secrets.json
and config.json
in cwd.External Party (no usage of Nomad configuration crate)
config.json
and secrets.json
in cwd (might not include all our networks).Just watcher:
UpdateHandler
(number of updates checked)UpdateHandler
(see ContractSync
for example)All agents that check home failure:
Add grafana display for the number of double updates a watcher has observed (fine if 0, go crazy with alerts if >= 1)
Add grafana graph for number of updates the watcher has inspected for double updates (should be up and to the right if watcher is functioning properly)
Add grafana display for number of times home has been observed as failed (fine if 0, go crazy with alerts if >= 1)
Add grafana graph for number of home failure checks an agent has performed (should be up and to the right if agent is healthy)
run_many
pattern and thus won't catch errors for channel faultsA declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.