ava-labs / avalanche-ops Goto Github PK

As a public project, it would benefit users to have a CONTRIBUTING.md doc that explains the nature of the project and how to contribute to it successfully. See https://github.com/operator-framework/rukpak/blob/main/CONTRIBUTING.md for an example of one.

Currently, we only tags VPC, ASG resources with user id, so for s3 bucket, it's hard to tell who created (and who forgot to clean up).

Use s3 put bucket tagging API to tag S3 buckets with sts user identity.

https://docs.avax.network/apis/avalanchego/apis/admin#adminaliaschain

We can get much higher TPS if gas fees are set right

[WORKER #25] failed transfer failed to send_transaction '(code: -32000, message: replacement transaction underpriced, data: None)'

Currently avalancheup-aws offers the ability to install a chain after using apply to spin up a set of validators. When it comes to deleting the chain however, delete only offers the ability to delete the entire network via the spec file. If a user wanted to delete a chain, but not delete the entire cluster, it could be useful. This could be done via adding to delete or making a new, standalone command.

• Static EBS volume provisioning #73
• Add more metrics

This is required for Hyper SDK dev net.

https://docs.avax.network/subnets/customize-a-subnet#chain-configs

The dev-box should be able to run custom scripts (ie send transactions) on startup.

Currently it's in my private repo...

Eliminate the need for SSH + whitelisted-subnets + systemctl restart

for aws infra provisioning, why was this in implemented in rust as opposed to using existing cloud provisioning tooling (terraform and/or ansible)?

First of all avalancheup-aws is a nice tool it helped me quickly setup nodes on AWS. While working on creating a new subnet and blockchain I noticed that the AvalancheGo version that was installed 1.10.0 was less than the version I needed 1.10,2. Once I pulled in a new binary and installed it things worked fine. Also during debug I updated the system unit file for AvalancheGo.

The issue is when the system gets migrated I assume due to spot workflow.

• I no longer have a systemd unit file for AvalancheGo
• The binary returns to the old version

I have not dug into this much but I assume this is related to how the state is managed in the spot instances. When a new VM is started it pulls the binary from s3 and probably the unit file as well.

A short section in the docs would be useful describing.

• how to upgrade AvalancheGo
• how to update managed files
• brief description of how spot works in this context.

ref. #29

Does avalanche-ops support building resources on Azure instead of AWS? Many customers, particularly in the financial sector, prefer to use Azure. Does it make sense to generalize the tool and support multiple clouds?

Useful to support more static anchor nodes for bootstrapping test network

We just need addresses for x/c chains

When using avalancheup-aws apply the end result is a list of endpoints and commands that can be run, for things like installing a new chain on the network. These commands are very helpful, but since they are only emitted to stdout, they can get lost in the terminal. I think writing the output of apply to stdout and a file (with the same aops- convention as the spec yaml file) could be helpful as a resource for users to retain access to all the auto-generated commands across terminals.

Use ava-labs/subnet-evm#357 to measure TPS

For instance,

# TYPE avalanche_2jo7eCZKHWG1qyYQ24SwBh8aB8h5bWhn7KCxYTk9PQTd2Tpzs_vm_eth_chain_txs_accepted counter
avalanche_2jo7eCZKHWG1qyYQ24SwBh8aB8h5bWhn7KCxYTk9PQTd2Tpzs_vm_eth_chain_txs_accepted 2971
# TYPE avalanche_2jo7eCZKHWG1qyYQ24SwBh8aB8h5bWhn7KCxYTk9PQTd2Tpzs_vm_eth_chain_txs_processed counter
avalanche_2jo7eCZKHWG1qyYQ24SwBh8aB8h5bWhn7KCxYTk9PQTd2Tpzs_vm_eth_chain_txs_processed 2971

thread 'tokio-runtime-worker' panicked at 'Integer overflow when casting to u64', /home/ubuntu/.cargo/registry/src/github.com-1ecc6299db9ec823/primitive-types-0.12.1/src/lib.rs:38:1

https://www.paradigm.xyz/2023/06/flood

1. Set up a pool of VPCs
2. Weekly trigger scalability testing job on subnet-evm
3. Reuse existing VPC to set up a custom network with subnet-evm (latest binaries)
4. Integrate install-subnet-chain command into avalancheup-aws apply so there's only a single command to install everything
5. Download install "flood"
6. Run against the custom network
7. Report results back

Just like rustup

#342 does some QoL linting but there are still some unresolved clippy warnings.

Ideal set of steps:

• Refactor or #[allow(...)] the code to have each warning accounted for
• Update CI to error in case of clippy warnings (see https://github.com/ava-labs/firewood/blob/2db03af7861f88066dd226eebf698fa001f310d3/.github/workflows/ci.yaml#L25 for an example).

After that we will have a fully linted codebase that is more production ready.

I would be nice to have a simple web UI or even a basic terminal only binary that manages the configuration files generated by avalancheup. This could allow for the configuration files to be managed in a centralized location and allow for easy tracking of which instances have been created/deleted.

MVP Functionality

• create
• list
• delete

dasconnor has been showcasing http://backstage.io/ perhaps that would work here if we decide to use a web UI.

Migrate https://github.com/ava-labs/avalanche-ops/blob/5314b5260802279c261c043c81a8a3bde32dd5d9/cdk/avalancheup-aws/cfn-templates/avalancheup.yaml to native CDK

• Specify subnet-evm git tag for auto downloads
• Specify VM ID for subnet-evm binary path
• Write some default chain config
• Customize upgrade bytes to write on each host
• Add existing subnet id to track-subnets flag

DEBUG[03-06|19:44:41.417] snow/engine/common/bootstrapper.go#224: Not enough votes received, restarting bootstrap... - Beacons: 2 - Failed Bootstrappers: 1 - bootstrap attempt: 85 (weight 200000000000000000, alpha 100000000000000001, failedBeaconWeight 100000000000000000)
DEBUG[03-06|19:39:06.881] snow/engine/common/bootstrapper.go#152: Received an Out-of-Sync GetAcceptedFrontierFailed - validator: 2bqPDyto7aHNfUvmKY7Ut2fnTgT9JvMJM - expectedRequestID: 2, requestID: 1
DEBUG[03-06|19:39:06.881] snow/networking/handler/handler.go#322: Finished handling sync message: get_accepted_frontier_failed
DEBUG[03-06|19:39:06.881] snow/networking/handler/handler.go#303: Forwarding sync message to consensus: (Op: get_accepted_frontier_failed, NodeID: NodeID-2bqPDyto7aHNfUvmKY7Ut2fnTgT9JvMJM, RequestID: 2)
DEBUG[03-06|19:39:06.881] snow/engine/common/bootstrapper.go#346: sent 2 more GetAccepted messages with 0 more to send
DEBUG[03-06|19:39:06.881] snow/networking/handler/handler.go#322: Finished handling sync message: get_accepted_frontier_failed
DEBUG[03-06|19:39:06.882] snow/networking/handler/handler.go#303: Forwarding sync message to consensus: (Op: accepted, NodeID: NodeID-AGzvo2xkGThWXu8JVYd9mVBqt2cf3iNM4, RequestID: 3, NumContainerIDs: 1)
DEBUG[03-06|19:39:06.882] snow/networking/handler/handler.go#322: Finished handling sync message: accepted
DEBUG[03-06|19:39:08.881] snow/networking/handler/handler.go#303: Forwarding sync message to consensus: (Op: get_accepted_failed, NodeID: NodeID-2bqPDyto7aHNfUvmKY7Ut2fnTgT9JvMJM, RequestID: 3)
DEBUG[03-06|19:39:08.881] snow/engine/common/bootstrapper.go#224: Not enough votes received, restarting bootstrap... - Beacons: 2 - Failed Bootstrappers: 1 - bootstrap attempt: 2 (weight 200000000000000000, alpha 100000000000000001, failedBeaconWeight 100000000000000000)

https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html

Just remove metrics collection service from avalanche-ops

When spinning up the dev-box in a region, currently the process to add the dev-box to the validator VPC is done manually. This could be automated as part of spinning up the dev-box. This would further enable the ability to run scripts automatically once the dev-box is created.

When running tools like avalancheup-aws frequently, many files get generated: spec files, prometheus rules, etc. These are all by default written to the user's home directory. It may make sense to move them into a separate hidden directory to not clutter the user's home directory, for example .aops or .avalanche-ops. Other avalanche tooling like avalanche-cli, avalanche-network-runner, and avalanchego all use a hidden home directory for their data.

This change would involve updating all the docstrings to point to the new directory.

Before

AWS_VOLUME_PROVISIONER_BIN_PATH=/home/ubuntu/volume-manager/target/release/aws-volume-provisioner
AWS_IP_PROVISIONER_BIN_PATH=/home/ubuntu/ip-manager/target/release/aws-ip-provisioner
AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH=/home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch

AVALANCHED_AWS_BIN_PATH=/home/ubuntu/avalanche-ops/target/release/avalanched-aws
AVALANCHEGO_BIN_PATH=/home/ubuntu/avalanchego/build/avalanchego

cd /home/ubuntu/avalanche-ops
/home/ubuntu/avalanche-ops/target/release/avalancheup-aws default-spec \
--arch-type amd64 \
--rust-os-type ubuntu20.04 \
--anchor-nodes 4 \
--non-anchor-nodes 4 \
--region ap-northeast-2 \
--instance-mode=on-demand \
--instance-size=4xlarge \
--ip-mode=elastic \
--metrics-fetch-interval-seconds 60 \
--ingress-ipv4-cidr 0.0.0.0/0 \
--upload-artifacts-aws-volume-provisioner-local-bin ${AWS_VOLUME_PROVISIONER_BIN_PATH} \
--upload-artifacts-aws-ip-provisioner-local-bin ${AWS_IP_PROVISIONER_BIN_PATH} \
--upload-artifacts-avalanche-telemetry-cloudwatch-local-bin ${AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH} \
--upload-artifacts-avalanched-aws-local-bin ${AVALANCHED_AWS_BIN_PATH} \
--upload-artifacts-avalanchego-local-bin ${AVALANCHEGO_BIN_PATH} \
--network-name custom \
--keys-to-generate 50 \
--enable-nlb \
--primary-network-validate-period-in-days 32

version: 2
id: aops-custom-202304-2TKPeY
aad_tag: avalanche-ops-aad-tag
resources:
  identity:
    account_id: '931867039610'
    role_arn: arn:aws:sts::931867039610:assumed-role/jumpcloud-experimental-developer/[email protected]
    user_id: AROA5R542S55AJJWVOGVF:[email protected]
  region: ap-northeast-2
  ingress_ipv4_cidr: 0.0.0.0/0
  s3_bucket: avalanche-ops-202304-42ziec8wgg-ap-northeast-2
  kms_symmetric_default_encrypt_key:
    id: 95c87189-0e73-4462-ba31-3befb680888c
    arn: arn:aws:kms:ap-northeast-2:931867039610:key/95c87189-0e73-4462-ba31-3befb680888c
  ec2_key_name: aops-custom-202304-2TKPeY-ec2-key
  ec2_key_path: /home/ubuntu/aops-custom-202304-2TKPeY-ec2-access.key
  cloudformation_ec2_instance_role: aops-custom-202304-2TKPeY-ec2-instance-role
  cloudformation_ec2_instance_profile_arn: arn:aws:iam::931867039610:instance-profile/aops-custom-202304-2TKPeY-instance-profile
  cloudformation_vpc: aops-custom-202304-2TKPeY-vpc
  cloudformation_vpc_id: vpc-0439425269f4ebc93
  cloudformation_vpc_security_group_id: sg-04b6f70bcf8cfa4df
  cloudformation_vpc_public_subnet_ids:
  - subnet-035f4501874063a4f
  - subnet-0ec6e08f0adbc5301
  cloudformation_asg_anchor_nodes:
  - aops-custom-202304-2TKPeY-anchor-amd64-01
  - aops-custom-202304-2TKPeY-anchor-amd64-02
  - aops-custom-202304-2TKPeY-anchor-amd64-03
  - aops-custom-202304-2TKPeY-anchor-amd64-04
  cloudformation_asg_anchor_nodes_logical_ids:
  - aops-custom-202304-2TKPeY-anchor-amd64-01
  - aops-custom-202304-2TKPeY-anchor-amd64-02
  - aops-custom-202304-2TKPeY-anchor-amd64-03
  - aops-custom-202304-2TKPeY-anchor-amd64-04
  cloudformation_asg_non_anchor_nodes:
  - aops-custom-202304-2TKPeY-non-anchor-amd64-01
  - aops-custom-202304-2TKPeY-non-anchor-amd64-02
  - aops-custom-202304-2TKPeY-non-anchor-amd64-03
  - aops-custom-202304-2TKPeY-non-anchor-amd64-04
  cloudformation_asg_non_anchor_nodes_logical_ids:
  - aops-custom-202304-2TKPeY-non-anchor-amd64-01
  - aops-custom-202304-2TKPeY-non-anchor-amd64-02
  - aops-custom-202304-2TKPeY-non-anchor-amd64-03
  - aops-custom-202304-2TKPeY-non-anchor-amd64-04
  cloudformation_asg_nlb_arn: arn:aws:elasticloadbalancing:ap-northeast-2:931867039610:loadbalancer/net/aops-custom-202304-2TKPeY-nlb/7dc1e1fb90b53789
  cloudformation_asg_nlb_target_group_arn: arn:aws:elasticloadbalancing:ap-northeast-2:931867039610:targetgroup/aops-custom-202304-2TKPeY-tg/5d50684ef30aac69
  cloudformation_asg_nlb_dns_name: aops-custom-202304-2TKPeY-nlb-7dc1e1fb90b53789.elb.ap-northeast-2.amazonaws.com
  cloudformation_asg_launch_template_id: lt-083db80a88191d265
  cloudformation_asg_launch_template_version: '1'
  cloudformation_ssm_install_subnet_chain: aops-custom-202304-2TKPeY-ssm-install-subnet-chain
  cloudwatch_avalanche_metrics_namespace: aops-custom-202304-2TKPeY-avalanche
  created_nodes:
  - kind: anchor
    machineId: i-004218e7af59e6e3c
    nodeId: NodeID-9Cf6ZXc5xUjfS63ZZY6Qe5PtUdDWEE2Eg
    proofOfPossession:
      publicKey: 0xae84eb755fb323debc330252978275886c5558254dd61f76b3036c1eb67d72811fb7849a40e9878f8022c47aed1bf479
      proofOfPossession: 0xb9e4a894161b54f25c8fb48b0fc181019728f92a78b263faaa6d711461cd2689b53c84b69c83a1258c3d4cf0c5224a6208490fa8f5f828a38b61296c7118c05503b35e16e967d7c
be23308afa2d211c1b5ceb8df797fbda492b66738010af26c
    publicIp: 43.200.185.164
    httpEndpoint: http://43.200.185.164:9650
  - kind: anchor
    machineId: i-0004f1cda22ee6d84
    nodeId: NodeID-AGYHGNq3CZjuZ54fsJdrPiPvtChKTejZr
    proofOfPossession:
      publicKey: 0xb12778e3660bfc2f76741784f1c1e76ba58894178c88db1340879964bd175cb0d7ff7e3050bc22060164a89a9223ff97
      proofOfPossession: 0xb654102321216e99686c7a0645093f48fecf7531efbbf402fda9810ada92782b9c6fe564555e3d947fb89f290a46850a0424f8a6f82636eea91bb8ea3b86ddd2fba79c3bf7216f5
eacd972419766e3c663fda293837e6f907888b2ab90435e43
    publicIp: 43.201.208.149
    httpEndpoint: http://43.201.208.149:9650
  - kind: anchor
    machineId: i-0779d859b49002dd8
    nodeId: NodeID-5veLTLisivPaHokQYZpb7usoYw52u6hD8
    proofOfPossession:
      publicKey: 0x96b5c7c898c61f30c24bcc611d1cdf211091b2ba030452e29f02952d218cfa0b7c31306a141407da21e566dfa4faf914
      proofOfPossession: 0x8eb7ce0d1c60256d6888cb1b8d6099f6a7a2f3f67fcaa856f8ab1dd775ebf6a2f79d6539e2d92d9e3449624791482a0a0baebebbb7ee1f7bb3bf65bcc9a4b4d3e1d061cad1dfea0
171d7893885ee546793e2b99f94b1ebe8d4659458ea0a5c36
    publicIp: 52.78.19.151
    httpEndpoint: http://52.78.19.151:9650

machine:
  anchor_nodes: 4
  non_anchor_nodes: 4
  arch_type: amd64
  rust_os_type: ubuntu20.04
  instance_types:
  - m5.4xlarge
  - c5.4xlarge
  instance_mode: on-demand
  ip_mode: elastic
  volume_size_in_gb: 300
avalanched_config:
  log_level: info
  use_default_config: false
enable_nlb: true
disable_logs_auto_removal: false
metrics_fetch_interval_seconds: 60
primary_network_validate_period_in_days: 32
prefunded_keys:
- key_type: hot
  private_key_cb58: PrivateKey-ewoqjP7PxY4yr3iLTpLisriqt94hdyDFNgchSxGGztUrTXtNN
  private_key_hex: 0x56289e99c94b6912bfc12adc093c9b51124f0dc54ac7a766b2bc5ccf558d8027
  addresses:
    1000000:
      x: X-custom18jma8ppw3nhx5r4ap8clazz0dps7rv5u9xde7p
      p: P-custom18jma8ppw3nhx5r4ap8clazz0dps7rv5u9xde7p
  short_address: 6Y3kysjF9jnHnYkdS9yGAuoHyae2eNmeV
  eth_address: 0x8db97C7cEcE249c2b98bDC0226Cc4C2A57BF52FC
  h160_address: 0x8db97c7cece249c2b98bdc0226cc4c2a57bf52fc

After

AWS_VOLUME_PROVISIONER_BIN_PATH=/home/ubuntu/volume-manager/target/release/aws-volume-provisioner
AWS_IP_PROVISIONER_BIN_PATH=/home/ubuntu/ip-manager/target/release/aws-ip-provisioner
AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH=/home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch

AVALANCHED_AWS_BIN_PATH=/home/ubuntu/avalanche-ops/target/release/avalanched-aws
AVALANCHEGO_BIN_PATH=/home/ubuntu/avalanchego/build/avalanchego

cd /home/ubuntu/avalanche-ops
/home/ubuntu/avalanche-ops/target/release/avalancheup-aws default-spec \
--arch-type amd64 \
--rust-os-type ubuntu20.04 \
--anchor-nodes 4 \
--non-anchor-nodes 4 \
--auto-regions 3 \
--instance-mode=on-demand \
--instance-size=4xlarge \
--ip-mode=elastic \
--metrics-fetch-interval-seconds 60 \
--ingress-ipv4-cidr 0.0.0.0/0 \
--upload-artifacts-aws-volume-provisioner-local-bin ${AWS_VOLUME_PROVISIONER_BIN_PATH} \
--upload-artifacts-aws-ip-provisioner-local-bin ${AWS_IP_PROVISIONER_BIN_PATH} \
--upload-artifacts-avalanche-telemetry-cloudwatch-local-bin ${AVALANCHE_TELEMETRY_CLOUDWATCH_BIN_PATH} \
--upload-artifacts-avalanched-aws-local-bin ${AVALANCHED_AWS_BIN_PATH} \
--upload-artifacts-avalanchego-local-bin ${AVALANCHEGO_BIN_PATH} \
--network-name custom \
--keys-to-generate 50 \
--enable-nlb \
--primary-network-validate-period-in-days 32

version: 3
id: aops-custom-202304-zeB7h6
aad_tag: avalanche-ops-aad-tag
resource:
  regions:
  - us-west-2
  - ap-northeast-2
  - eu-west-1
  s3_bucket: avalanche-ops-202304-42ziec8wgg-us-west-2
  ingress_ipv4_cidr: 0.0.0.0/0
  regional_resources:
    us-west-2:
      region: us-west-2
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.us-west-2.key
    eu-west-1:
      region: eu-west-1
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.eu-west-1.key
    ap-northeast-2:
      region: ap-northeast-2
      ec2_key_name: aops-custom-202304-zeB7h6-ec2-key
      ec2_key_path: /home/ubuntu/aops-custom-202304-zeB7h6-ec2-access.ap-northeast-2.key
machine:
  total_anchor_nodes: 4
  total_non_anchor_nodes: 4
  arch_type: amd64
  rust_os_type: ubuntu20.04
  instance_mode: on-demand
  ip_mode: elastic
  volume_size_in_gb: 300
  regional_machines:
    eu-west-1:
      anchor_nodes: 2
      non_anchor_nodes: 2
      instance_types:
      - c6a.4xlarge
      - m6a.4xlarge
      - m5.4xlarge
      - c5.4xlarge
    us-west-2:
      anchor_nodes: 1
      non_anchor_nodes: 1
      instance_types:
      - c6a.4xlarge
      - m6a.4xlarge
      - m5.4xlarge
      - c5.4xlarge
    ap-northeast-2:
      anchor_nodes: 1
      non_anchor_nodes: 1
      instance_types:
      - m5.4xlarge
      - c5.4xlarge
upload_artifacts:
  avalanched_local_bin: /home/ubuntu/avalanche-ops/target/release/avalanched-aws
  aws_volume_provisioner_local_bin: /home/ubuntu/volume-manager/target/release/aws-volume-provisioner
  aws_ip_provisioner_local_bin: /home/ubuntu/ip-manager/target/release/aws-ip-provisioner
  avalanche_telemetry_cloudwatch_local_bin: /home/ubuntu/avalanche-telemetry/target/release/avalanche-telemetry-cloudwatch
  avalanchego_local_bin: /home/ubuntu/avalanchego/build/avalanchego
  prometheus_metrics_rules_file_path: /home/ubuntu/aops-custom-202304-zeB7h6-prometheus-metrics-rules.yaml
avalanched_config:
  log_level: info
  use_default_config: false
enable_nlb: true
disable_logs_auto_removal: false
metrics_fetch_interval_seconds: 60
primary_network_validate_period_in_days: 32
prefunded_keys:
- key_type: hot

It's possible to optimize the Rust CI by caching the cargo home directory, or specifically certain files under .cargo, and then sharing those files between different CI jobs. This avoids the need to re-download dependencies during multiple jobs.

See https://doc.rust-lang.org/cargo/guide/cargo-home.html#caching-the-cargo-home-in-ci for more details.

This could potentially be done for all of our Rust projects. Not sure which have this implemented already.