harmony-one / harmony-ops

Harmony Ops Master Repository.

Harmony Foundational Node Success Guide

This provides information for Harmony Foundational Nodes to be sucessful.

What is Harmony?

Our vision is ‘Open Consensus for 10 Billion People.’

Our open infrastructure is a high-throughput, low-latency, and low-fee consensus platform designed to power decentralized economies of the future.

We aim to deliver both scalability and decentralization. The promise of open consensus is to enable decentralized coordination at scale but no platform has yet been able to achieve both.

Similar to the way that Google vertically integrates its search infrastructure, we take a full stack approach to solve consensus at scale. We apply 10x innovations at every layer in consensus algorithms, systems, and networking to maximize the performance of our network while maintaining decentralization. Our end-to-end integration allows us to iterate faster and make more aggressive optimizations than could be done with a modular approach.

Our Approach

Harmony's technical architecture implements a full & secure sharding scheme, efficient consensus, and scalable networking infrastructure.

Harmony relies on a secure sharding process, where validators are distributed into shards based on a randomness that is both unpredictable and unbiasable. Harmony is a Proof-of-Stake blockchain, where validators needs to stake a certain amount of tokens to be eligible for block validation. Harmony integrates an efficient consensus protocol called FBFT that combines BLS multi-signature and view change protocol to achieve high robustness and low latency. With the adoption of networking technology including RaptorQ fountain code and Kademlia routing, Harmony is able to achieve cross-shard transactions that scale sub linearly by the number of shards.

By innovating on each layer, Harmony aims to provide a scalable, secure, and decentralized system that supports economic activities, including data marketplaces, gaming, and financial transactions, for billions of people.

To participate in our research and technical discussions, join our Research forum on talk.harmony.one

harmony-ops's Issues

Upgrade All 4 Explorers in the Miannet

Per RJ' request to fix the explorer 2 stuck issue.

This ticket is related to harmony-one/harmony#1575

DevOps.sh Script improvement

For log checking and Bingo keyword verification, the current script works well for the legacy running nodes, but it won't work for node launched by Terraform due to the location of log folder change. I am thinking to improve it.

@LeoHChen Your thoughts?

Restart Explorer Backend Service on Sep 12, 2019

Enable Travis CI to VDF Repo

Upgrade Grafana EBS Volume to 250 GB

Per LC's request, we'd like to store one month of historical data on Grafana server. In that case, we need to increase the size of the Grafana to 250 GB.

Amazon EC2 Instance Retirement

5 in total, they are:

(instance-ID: i-02da31bb79757709b) in the us-west-2 region
(instance-ID: i-00d3acf938c7f8520) in the us-west-2 region
(instance-ID: i-058e4758c92d0a955) in the us-west-2 region
(instance-ID: i-091232c0b9f84ef02) in the us-west-2 region
(instance-ID: i-0980e4873f3764c84) in the us-west-2 region
(instance-ID: i-08b896007684d3611) in the eu-west-1 region

Recover Explorer 2 in the Mainnet on Sep 11, 2019

From the explorer website, the explorer for shard 2 is offline, but its status is online on the /1h website.

Consolidated Issues for Harmony Explorer

Priority: Low

URL: https://explorer.harmony.one/#/

All 4 shards are working properly at the moment, but the information displayed in the explorer website is not accurate.

From the messages gathered from Discord and Wechat, there are a few issues we'd like to address:

• stability issue, it went offline sometime for no specific reasons
• it shows 2 shards are online, but all 4 are good
• number of block count get reset, and sometimes it needs to re-sync

Recover an Offline Node on Sep 7, 2019

https://harmonyone.pagerduty.com/incidents/POL952U

The monitor shard-3-008 (13.228.72.14) is currently DOWN (Port Is Not Listening).

We were not able to recover the node using the devops.sh script, then we decide to launch a new node to replace the old one using Terraform.

Finally

• update uptime

./uptimerobot.sh -G update shard-3-008 54.249.98.66 3

• update shard3.txt (local on devops instance, and on S3)

One click deployment to GCP

Use https://github.com/harmony-one/HarmonyFoundationalNode

To deploy the Harmony node on GCP using Terraform.

Verify Harmony Node Creation on Vultr

Priority: LOW

Verify Harmony node running tool on Vultr.

https://github.com/AndyBoWu/HarmonyVultrNode

Enable WSS Protocol for api.{s0, s1, s2, s3}.t.hmny.io

This is for the Mainnet, and need to create each endpoint for each shard.

Investigation needed for the shrinking disk space issue on the Grafana Server

This issue happened before about 2 months ago since when we started to monitor 400+ nodes in our testnet. The issue is about Prometheus service keeps all historical data from every single server, and there is no mechanism yet to recycle those data.

Need to figure out a decent solution to solve this problem.

Pager duty integration with Uptime

Setup pager on 6 nodes down, using pager duty.

Need to add a parameter to node.sh to force download the binaries

We are trying to restart shard 2 but noticed that a few nodes can not be restarted due to some corrupted binaries.

Per LC's suggestion, might be better to add a parameter to force a node to download the binaries in the node.sh script.

Monitor is DOWN: shard-1-077

Working on it.

Update Runbook to launch Replacement Nodes

Update run book about how to launch new nodes using Terraform, also need to do at least once to verify its procedure.

Update Shard 2 leader IP and Online Status

Recover Offline Nodes on Sep 7, 2019

INCIDENT #861

Enable Reverse proxy to http://grafana.harmony.one

Requirements:
1/ to hide the port :3000
2/ enable https

Need to launch two terraform nodes to replace two degraded nodes

Both are located in N.California.

i-0a44e74caadf4898c and i-0c00e6abd5e150327

Scale Up DevOps Instance on Sep 10, 2019

Wallet processes are consuming too much system resouce on Devops instance.

Need to upgrade the instance to m5a.

Extend EBS Volume to All Nodes in the Mainnet

Enable WSS Protocol for api.{s0, s1}.b.hmny.io

Per the request from the Math wallet team.

http://explorer2.harmony.one/ Does not Load at all

One of the possible reasons is that the SSL certificate has expired.

Recover an offline node explorer-3

Explorer for shard 3 is down from this url: https://explorer.harmony.one/#/

local diagnostic system for foundational node runners

we need to design a system to help foundation node runners to run/monitor their nodes properly. This will also help reduce our support task, and enhance our network liveness.

The goals:

1. monitor the running status of FN nodes, earning, issue, online/offline, alert/paging
2. monitor the status of the host, cpu, ram, networking, disk, and alarm
3. collect diagnostic logs to help our team triage the issues of the node, reduce communication cost

The project can be implemented in different stages. We shall also use existing framework/system to accomplish the tasks.

Stages

In the interim stage, we can employ a monitoring script and a diagnostic script.
In the longer term, we can build local grafana server, logstash, and pushagent, to collect the metrics from node, from local host, and present/visualize the data to make it more user-friendly.

Monitoring script

we have a few monitoring script already. We need to adapt it to the latest log.
ETA: 10 hours

Diagnostic script

Node runner can run this script to collect issues, if they report they are not earning or offline, or not working. It can create a tarball and upload the data to a public repo so that we can do further analysis by the team.
ETA: 20 hours

Notification of Amazon S3 buckets configured for public access

Restrict access permission to the following buckets:

harmony-protocol
unique-bucket-bin
devnet.harmony.one
haochen-harmony-pub
pub.harmony.one
testnet.harmony.one

Enable SSL Certificate to http://grafana.harmony.one

Might need to enable SSL certificate to this domain in order to deliver a professional and commercial website.

nodes.harmony.one update

The content in this page are links to our Google docs, which is not professional. Those has to be fixed. Please convert them to markdown format.
https://nodes.harmony.one/foundational-node-playbook/guides

Please work with @john-harmony to get it done. Thanks.

Update Target List for Grafana Monitoring

we have been adding new nodes to the mainnet, but the target list has not been updated.

Prometheus and Grafana deployment using Terraform

We have a bash script for this task, but in the long term, we should provide this server using an open-source IaC tool. In this case, Terraform is the best candidate for this project based on our research.

Tools Needed to be installed:

• Prometheus
• Grafana
• Prometheus Alertmanager
• Prometheus Pushgateway?

Recover Offline Nodes on Sep 4, 2019

5 in total

52.211.227.212
63.32.95.254
34.244.236.58
34.250.59.78
34.254.226.205

Shard 0..2 showing Offline in /1h and /balances Websites

Happened before, the root cause is that the IP addresses of leaders got changed. I can fix the issue again, but the real question is how often does the leader got rotated. And, how can we automate this process.

Create HTTPS Pipeline for the API/Pangaea/Mainnet Endpoints

john_whitton | HarmonyToday at 11:56 AM
@andy | Harmony https for these
@eric Yu - 麦子钱包 Thanks for the hard work and the updates - looking forward to seeing the next version tomrrow :)
Here are the URL's for all three environments (also in the spreadsheet above)
Mainnet - http://s1.t.hmny.io:9500/
Pangaea - http://s1.pga.hmny.io:9500/
Harnony Betanet: - http://s1.b.hmny.io:9500/

Shard 3 is OFFLINE on /1h Website - Sep 9, 2019

Recover offline nodes on Sep 8, 2019

https://harmonyone.pagerduty.com/incidents/PZQFBXA

Monitor is DOWN: shard-1-098

Recover 3 Offline Nodes on Aug 31

Status: https://status.harmony.one

shard-1-044
shard-2-101
shard-3-105

Consolidated Issues for /1h Website

ISSUE 1
Shard 2 shows offline, but I suspect the leader has been rotated.

ISSUE 2
The hourly reward does not look right, it should be somewhere around 46.

Send out notification to offline node runners

Been doing technical supports for our node runners, one common request is that the offline node runners would like to have a prompt (email or sms) notification about offline nodes.

Add documentation about how to restart each shard properly

Need to add this documentation in the DevOps Runbook

MainNet Preparation

This Issue keeps track of tasks that may be worked on in preparation for mainnet.

Task List

• Monitoring
  • mystatus.sh
  • uptimeRobot automatic scripting
• Documentation
  • Create Harmony Wiki
    • Status Page
  • Trouble Shooting Playbook
  • Foundational Node Operators Guide
  • Release Process

Some nodes earning rewards less than expected

Some customer complained about his nodes are earning rewards around 27 per hour, but it should be around 40. Need to investigate this issue.

Note that the customer has shared the account addresses with me.

EC2 Instance Retirement - i-0f06451ba2ca27c9c

EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-0f06451ba2ca27c9c) associated with your AWS account (AWS Account ID: 656503231766) in the eu-central-1 region. Due to this degradation your instance could already be unreachable. We will stop your instance after 2019-09-11 08:00 UTC.

Shard 0..2 are offline from /balances website

In the /balances webpage, it shows shard 0, 1 and 2 are offline, but from the explorer webpage, all 4 shards are up and online. Very likely the IP of the leader in those shards (0, 1, and 2) have been changed.

Incorrect Info for Shard 3

Shard 3 is on Block . Status is: OFFLINE... (Last updated: Wed Sep 4 00:00:00 UTC 2019)

Finish the Post-Mortem Report for the Friday Outage (Aug 30, 2019)

Per LC's request.

Decommission Retired Grafana Server

There was an issue about the underlying hardware running Grafana server. We launched a new server to replace the old one. Need to delete the old retired grafana server under 6565-0323-1766 AWS account.

Harmony Node Reboot Recovery

We need to automatically recovery harmony node after ec2 instance reboot.

1. add a service script to auto start soldier after boot up.
2. modify userdata script to install the service and run the service the first time
3. after soldier start, run local curl command to re-init, if there is an existing init.json file.

Add the documentation about how to restart Terraform Nodes

into the DevOps Runbook