An easy method to configure public and private monitoring for Coti nodes. This setup is compatible with nodes installed with Systemd or Docker.
Click to see a live example of a dashboard produced from this setup.
This method provides:
- A Grafana dashboard visualisation system
- Automatic SSL certificate management when needed
- Server monitoring and health statistics with Prometheus
- Log tracking and querying with Loki
This installation guide produces working setups for nodes installed both with systemd (a.k.a. GeordieR's installation scripts) and with docker (for example this Coti-Docker installation guide), although the instructions for each installation type differ at certain points.
If docker
and docker-compose
are not already installed on your system, you will need to run the following commands to install them:
sudo su
curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
curl -L https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose
Check whether the installations were successful using
docker --version
docker-compose --version
For users whose nodes are installed with Docker, the monitoring system is set up to be accessed from the url https://monitoring.<your-node-url>
. If you have not set up a subdomain record with your DNS provider, it is likely you will need to do this to make that url accessible.
To do this, you can either add a wildcard subdomain (*
), or the specific subdomain you intend to set up (monitoring.
). In my case I went with a wildcard subdomain, and my working DNS configuration looks like
I suggest having your Coti node located under the directory ~/coti-node
. We will install the monitoring setup alongside that directory, under ~/coti-node-monitoring
. From your home directory, run
git clone https://github.com/tomjwells/coti-node-monitoring.git && cd coti-node-monitoring
The .env
file defines important environment variables used to set up the monitoring. Create a .env
file in the coti-node-monitoring
directory and enter your environment variables in the following format:
SERVERNAME="<your-node-domain.tld>"
GRAFANA_USERNAME="<Enter a username here>"
GRAFANA_PASSWORD="<Enter a password here>"
where
SERVERNAME
is the web address of your node, excludinghttp(s)://
andwww.
, for examplesubdomain.your-node-domain.tld
,GRAFANA_USERNAME
is the username you wish to use for logging in to Grafana, and,GRAFANA_PASSWORD
is the password you wish to use for logging in to Grafana.
You may make up your own username, or use the email associated with your Coti node. For the password, any long string that is difficult to guess will be suitable.
If you followed the Coti-Docker installation guide, you will already have a network called gateway
running. You can check the Docker networks with docker network ls
. If your node is running with systemd, or the network was not created somehow, you can create the gateway
network with
docker network create --driver=bridge --attachable --internal=false gateway
Check whether the Docker network has been created with docker network ls
. If you see the gateway
network in the output, then you are ready to continue.
Now you are ready to run the monitoring stack! If your node is installed with Docker, run
docker-compose up
This pulls the monitoring software for you and launches it once it is downloaded. If everything goes successfully, you are done.
First we need to add some paths to the web server configuration. I have included a script which makes the necessary changes for you. Make sure you are logged in as the root user with sudo su
. Then, the script can be run with
./configure-webserver.sh
Now you are ready to run the monitoring stack! If your node is running with systemd, run
docker-compose -f docker-compose-systemd.yml up
This pulls the monitoring software for you and launches it once it is downloaded. If everything is successful, you are done.
Grafana usually takes between 10-30 seconds to become ready, so after some seconds, navigate in your browser to monitoring.<your-node-url>
(Docker installs) or <your-node-url>/monitoring
(Systemd installs). If everything is working, you will see the Grafana sign-in page:
Use the sign in credentials you set in the .env
file, GRAFANA_USERNAME
and GRAFANA_PASSWORD
, to log in.
If you see the following welcome screen:
then congrats, you did it!
๐
I wish to cover some of the features that come out-of-the-box with this installation method. These could be improved and extended further over time and I would welcome improvements and suggestions from others.
Out of the box features
- Dashboards:
Coti Public Dashboard
- a publicly sharable dashboard. (Example)Coti Private Dashboard
- a dashboard which has a bit more information
- Grafana Datasources:
While the dashboards are immediately useful and give lots of information about the state of your node, the preconfigured Grafana datasources allow you to create your own custom queries and graphs.
Take down a single container:
docker-compose rm -sv <container_name>
Start a single container:
docker-compose up <container_name>
Restart an individual container:
docker restart <container_name>
Follow logs of a single service/container:
docker-compose logs <container_name> --follow
This section will be used to list solutions common debugging problems related to this installation process.
I get HTTPS errors or strange connectivity problems even though everything is set up correctly, or `postgres: could not get migration log" error="failed to check table existence: dial tcp 172.19.0.7:5432: connect: connection refused"`
In creating this setup, I found that sometimes I would get intermittent problems with networking between docker containers. In debugging, I found that destroying and recreating the `gateway` network would fix this. My suspicion is this is a slight bug/incompatability in Docker, but I am not 100% clear about the cause.Whenever I had these issues, I was able to solve them by running
- docker network rm gateway && docker network create --driver=bridge --attachable --internal=false gateway
This installation method is stable and works well in my tests, but there is plenty of room for improvement. I have many ideas that have not been explored. Some of these are
- Fixing any bugs
- Configuring alerts (e.g. based on RAM usage, CPU usage, and response times)
- More sophisticated dashboards that take better advantage of the unique information available from Coti nodes
- Monitoring traefik (the web server) and charting response times
Dashboards and panels are very easy to contribute, as they can simply be exported from Grafana as JSON files, and all that's needed to make them appear is place them in the directory config/grafana/provisioning/dashboards
.
If you are interested in contributing to any of these, I would happily take suggestions or code submissions, or make this repository accessible to collaborators.
- Credits to the Coti community for the vital support and guidance given to testnet and mainnet node operators.
- Credits to all the organizations who contribute open-source software, making all this possible for free. Namely Docker, Grafana and Traefik.
Stay Coti. ๏ธโ๐ฅ