coti-node-monitoring

Coti Node Monitoring

An easy method to configure public and private monitoring for Coti nodes. This setup is compatible with nodes installed with Systemd or Docker.

Click to see a live example of a dashboard produced from this setup.

This method provides:

A Grafana dashboard visualisation system
Automatic SSL certificate management when needed
Server monitoring and health statistics with Prometheus
Log tracking and querying with Loki

Installation Instructions

This installation guide produces working setups for nodes installed both with systemd (a.k.a. GeordieR's installation scripts) and with docker (for example this Coti-Docker installation guide), although the instructions for each installation type differ at certain points.

If docker and docker-compose are not already installed on your system, you will need to run the following commands to install them:

sudo su
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
curl -L https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose

Check whether the installations were successful using

docker --version
docker-compose --version

DNS Settings (Docker installated nodes only)

For users whose nodes are installed with Docker, the monitoring system is set up to be accessed from the url https://monitoring.<your-node-url>. If you have not set up a subdomain record with your DNS provider, it is likely you will need to do this to make that url accessible.

To do this, you can either add a wildcard subdomain (*), or the specific subdomain you intend to set up (monitoring.). In my case I went with a wildcard subdomain, and my working DNS configuration looks like

1. Clone the Repository

I suggest having your Coti node located under the directory ~/coti-node. We will install the monitoring setup alongside that directory, under ~/coti-node-monitoring. From your home directory, run

git clone https://github.com/tomjwells/coti-node-monitoring.git && cd coti-node-monitoring

2. Create a `.env` File

The .env file defines important environment variables used to set up the monitoring. Create a .env file in the coti-node-monitoring directory and enter your environment variables in the following format:

SERVERNAME="<your-node-domain.tld>"
GRAFANA_USERNAME="<Enter a username here>"
GRAFANA_PASSWORD="<Enter a password here>"

where

SERVERNAME is the web address of your node, excluding http(s):// and www., for example subdomain.your-node-domain.tld,
GRAFANA_USERNAME is the username you wish to use for logging in to Grafana, and,
GRAFANA_PASSWORD is the password you wish to use for logging in to Grafana.

You may make up your own username, or use the email associated with your Coti node. For the password, any long string that is difficult to guess will be suitable.

3. Create the `gateway` network

If you followed the Coti-Docker installation guide, you will already have a network called gateway running. You can check the Docker networks with docker network ls. If your node is running with systemd, or the network was not created somehow, you can create the gateway network with

docker network create --driver=bridge --attachable --internal=false gateway

Check whether the Docker network has been created with docker network ls. If you see the gateway network in the output, then you are ready to continue.

🏃 Running the Monitoring Stack

Docker Installed Nodes

Now you are ready to run the monitoring stack! If your node is installed with Docker, run

docker-compose up

This pulls the monitoring software for you and launches it once it is downloaded. If everything goes successfully, you are done.

Systemd Installed Nodes

First we need to add some paths to the web server configuration. I have included a script which makes the necessary changes for you. Make sure you are logged in as the root user with sudo su. Then, the script can be run with

./configure-webserver.sh

Now you are ready to run the monitoring stack! If your node is running with systemd, run

docker-compose -f docker-compose-systemd.yml up

This pulls the monitoring software for you and launches it once it is downloaded. If everything is successful, you are done.

Using Grafana

Logging In

Grafana usually takes between 10-30 seconds to become ready, so after some seconds, navigate in your browser to monitoring.<your-node-url> (Docker installs) or <your-node-url>/monitoring (Systemd installs). If everything is working, you will see the Grafana sign-in page:

Use the sign in credentials you set in the .env file, GRAFANA_USERNAME and GRAFANA_PASSWORD, to log in.

If you see the following welcome screen: then congrats, you did it!

🎉

Features

I wish to cover some of the features that come out-of-the-box with this installation method. These could be improved and extended further over time and I would welcome improvements and suggestions from others.

Out of the box features

Dashboards:
- Coti Public Dashboard - a publicly sharable dashboard. (Example)
- Coti Private Dashboard - a dashboard which has a bit more information
Grafana Datasources:
- Prometheus
- Loki

While the dashboards are immediately useful and give lots of information about the state of your node, the preconfigured Grafana datasources allow you to create your own custom queries and graphs.

Useful Docker Management Commands

Take down a single container:

docker-compose rm -sv <container_name>

Start a single container:

docker-compose up <container_name>

Restart an individual container:

docker restart <container_name>

Follow logs of a single service/container:

docker-compose logs <container_name> --follow

🧑‍💻 Debugging

This section will be used to list solutions common debugging problems related to this installation process.

I get HTTPS errors or strange connectivity problems even though everything is set up correctly, or `postgres: could not get migration log" error="failed to check table existence: dial tcp 172.19.0.7:5432: connect: connection refused"`

In creating this setup, I found that sometimes I would get intermittent problems with networking between docker containers. In debugging, I found that destroying and recreating the `gateway` network would fix this. My suspicion is this is a slight bug/incompatability in Docker, but I am not 100% clear about the cause.
Whenever I had these issues, I was able to solve them by running

docker network rm gateway && docker network create --driver=bridge --attachable --internal=false gateway

which recreates the network.

Contributing

This installation method is stable and works well in my tests, but there is plenty of room for improvement. I have many ideas that have not been explored. Some of these are

Fixing any bugs
Configuring alerts (e.g. based on RAM usage, CPU usage, and response times)
More sophisticated dashboards that take better advantage of the unique information available from Coti nodes
Monitoring traefik (the web server) and charting response times

Dashboards and panels are very easy to contribute, as they can simply be exported from Grafana as JSON files, and all that's needed to make them appear is place them in the directory config/grafana/provisioning/dashboards.

If you are interested in contributing to any of these, I would happily take suggestions or code submissions, or make this repository accessible to collaborators.

✨ Credits

Credits to the Coti community for the vital support and guidance given to testnet and mainnet node operators.
Credits to all the organizations who contribute open-source software, making all this possible for free. Namely Docker, Grafana and Traefik.

STAY COTI

Stay Coti. ️‍🔥

tomjwells / coti-node-monitoring Goto Github PK