intergral / observability-agent

Wrapper for the Grafana Agent to automatically configure exporters and ship to FusionReactor

Introduction

The Observability Agent autoconfiguration and installer tool is a wrapper for Grafana Alloy that can install Alloy, detect which services are running on your machine, and automatically create a configuration file with integrations for detected services. A Node/Windows exporter integration will be added by default.

Prerequisites

For Unix machines, the script will have to be run with root privileges, otherwise you will be prompted for your password during execution. Windows machines must have Powershell 5.1 or later installed. macOS is not currently supported.

The script will automatically detect what's running on your machine and add integrations to the config file. For this, each service you want to have an integration for must be running on its default port, these are:

Integration	Default Port
MySQL	3306
MSSQL	1433
Postgres	5432
RabbitMQ	5672
RabbitMQ Exporter	15692
Redis	6379
Kafka	9092
Elasticsearch	9200
Mongo	27017
OracleDB	1521

Procedure

Both the Static Mode installer and Flow Mode installer have been deprecated and superseded by Grafana Alloy installer

Latest Release

Linux

To download and run the installer, in a terminal, run:

curl -O -L "https://github.com/intergral/observability-agent/releases/download/v0.2.3/observability-agent-autoconf.sh"
chmod a+x "observability-agent-autoconf.sh"
sudo /bin/bash observability-agent-autoconf.sh

Windows

To download and run the installer, open powershell admin terminal, navigate to your desired download folder and run:

Invoke-WebRequest -Uri "https://github.com/intergral/observability-agent/releases/download/v0.2.3/observability-agent-autoconf.ps1" -OutFile "observability-agent-autoconf.ps1"
.\observability-agent-autoconf.ps1

The installer for windows assumes you are installing Grafana Alloy in the default location on the C drive
This is required for the config file to be placed in the correct location for the Grafana Alloy to read it

Options

Grafana Alloy installation is enabled by default. To run without installing Alloy, add --install false to the end of the run command. For example:
sudo path/to/observability-agent-autoconf.sh --install false

To modify a pre-existing config file, add --config.file, followed by the path to the file, to the end of the run command. For example: sudo path/to/observability-agent-autoconf.sh --config.file path/to/configfile
A backup of your original file will be created

To disable prompts that wait for user input, add --prompt false as command arguments.

You can use both --install and --config.file options in the same run command, order is irrelevant. For example:
sudo path/to/observability-agent-autoconf.sh --install false --config.file path/to/config
or
sudo path/to/observability-agent-autoconf.sh --config.file path/to/config --install false

For Windows, you can add --disable-dl-progress-bar true to potentially speed up downloads.

Docker

When running in Docker, you will not be prompted for any information. Therefore, you must specify an api key before running. Additionally, you must set the relevant environment variables for whichever services you have running, so they can be configured. These environment variables can be found in the Environment Variables section.

To run in docker, we provide prebuilt images. See our Docker Hub repository for more information

Environment Variables

To add integrations without being prompted for credentials, there are several environment variables you can use:

Ingest

Variable	Type	Description
api_key	string	API Key to authenticate with your FusionReactor Cloud Account
metrics_Endpoint	string	Default: https://api.fusionreactor.io/v1/metrics
logs_Endpoint	string	Default: https://api.fusionreactor.io/v1/logs

Metric Exporters

Variable	Type	Description
mysql_user	string	User for the local Mysql database
mysql_password	string	Password for the local Mysql database
mysql_disabled	bool	Enables/Disables the Mysql exporter (enabled by default)
mssql_user	string	User for the local Mssql database
mssql_password	string	Password for the local Mssql database
mssql_disabled	bool	Enables/Disables the Mssql exporter (enabled by default)
postgres_user	string	User for the local Postgres database
postgres_password	string	Password for the local Postgres database
postgres_db	string	Database name for the local Postgres database (defaults to username)
postgres_disabled	bool	Enables/Disables the Postgres exporter (enabled by default)
rabbitmq_disabled	bool	Enables/Disables the RabbitMQ exporter (enabled by default)
rabbitmq_instance_label	string	Optional variable to set the RabbitMQ instance identifier
redis_disabled	bool	Enables/Disables the Redis exporter (enabled by default)
elasticsearch_user	string	User for the Elastic search instance
elasticsearch_password	string	Password for the Elastic search instance
elasticsearch_disabled	bool	Enables/Disables the Elastic search exporter (enabled by default)
mongodb_user	string	User for the local Mongo database
mongodb_password	string	Password for the local Mongo database
mongodb_disabled	bool	Enables/Disables the MongoDB exporter (enabled by default)
oracledb_user	string	User for the local Oracle database
oracledb_password	string	Password for the local Oracle database
oracledb_disabled	bool	Enables/Disables the OracleDB exporter (enabled by default)

Exporting metrics from external machines

To replace these with a custom connection string, there are several environment variables you can use:

Variable	Type	Example (Defaults)
mysql_connection_string	string	<username>:<password>@(<host>:3306)/
mssql_connection_string	string	sqlserver://<username>:<password>@<host>:1433
postgres_connection_string	string	postgresql://<username>:<password>@<host>:5432/shop?
rabbitmq_scrape_target	string	<host>:15692
redis_connection_string	string	<host>:6379
kafka_connection_string	string	["<host>:9092"]
elasticsearch_connection_string	string	http://<username>:<password>@<host>:9200
mongodb_connection_string	string	mongodb://<username>:<password>@<host>:27017/
oracledb_connection_string	string	oracle://<username>:<password>@<host>:1521/ORCLCDB

RabbitMQ requires an internal exporter to be enabled. Visit the documentation for more information.

Log Exporters

If you wish to enable log collection, the following environment variables must be set:

Variable	Type	Example	Description
log_collection	bool	true	Enables log collection
service_name	string	service	Set a name for you log collection service
log_path	string	/service/logs	Set a file path for your log collection service

Open Telemetry

If you wish to enable Open Telemetry metrics and traces, the following environment variables must be set:

Variable	Type	Example	Description
otel_collection	bool	true	Enables Open Telemetry metrics and traces

Scraping from additional exporters

At present, there are some integrations we don't support out the box. Use of these integrations is via a scrape endpoint. To scrape these exporters, you can use the following environment variables to set a list of exporters and their corresponding endpoints to be scraped. The list must be wrapped in double quotes and each value must be separated by a comma and a space.

Variable	Type	Example	Description
scrape_targets	string	"nginxexporter:9113, iisexporter:1234"	List of endpoints for the exporters

.Env Files

If you wish to use an environment file to set environment variables, rather than setting them as system environment variables, you can do so by naming the file ".env" and placing it in the same directory as the "observability-agent-autoconf" script.

Example ".env" file:

api_key=1234567890
mysql_connection_string=root:my-secret-pw@(mysql:3306)/
log_collection=true
service_name=service
log_path=path

observability-agent's Issues

Enable Grafana service to ensure it starts when the system starts

Migrate Observability Agent to Grafana Alloy

Suggestion: Install script should cleanup/delete downloaded Grafana-Agent files

The linux installer downloads files for the Grafana-Agent install. This file is left behind.

You can attempt to delete the file on exit using trap EXIT

Windows: Pin version of grafana-agent

Suggestion: Make use of variables to make script more concise

This is personal preference, so dismiss if you want since it's subjective and you're the maintainers.

For integrations that don't require a password, you can use shell parameter expansion to reduce the amount of lines needed.

# Detect Kafka
if { (ss -ltn | grep -qE :9092) || [ -n "${kafka_connection_string}" ]; } && [ "${kafka_disabled}" != true ]; then
  echo "Kafka detected"
  kafka_conn_string=${kafka_connection_string:='127.0.0.1:9092'}
    cat <<EOF >> "$CONFIG"
prometheus.exporter.kafka "example" {
  kafka_uris = ["${kafka_conn_string}"]
}

prometheus.scrape "kafka" {
  targets    = prometheus.exporter.kafka.example.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

EOF
  echo "Kafka integration enabled"
fi

For integrations that require a password, you can do your connection string retrieval/parsing up front to reduce the length of branching if statements.

# Detect MySQL
if { (ss -ltn | grep -qE :3306) || [ -n "${mysql_connection_string}" ]; } && [ "${mysql_disabled}" != true ]; then
  echo "MySQL detected"
  conn_string=$mysql_connection_string
  # Check if connection string already set in environment
  if [ -z "${conn_string}" ]; then
    # Check if credentials already set in environment
    if [ "${mysql_user}" ] && [ "${mysql_password}" ]; then
      echo "MySQL credentials found";
      conn_string="$mysql_user:$mysql_password@(127.0.0.1:3306)/"
    # Prompt user for credentials
    elif [ "$PROMPT" != false ]; then
      echo "MySQL credentials not found";
      while true; do
          echo "Enter your username:"
          read -r user
          if [ -z "$user" ]; then
              echo "Username cannot be empty. Please enter a valid username."
          else
            break
          fi
      done

      while true; do
          echo "Enter your password:"
          read -rs pass
          if [ -z "$pass" ]; then
              echo "Password cannot be empty. Please enter a valid password."
          else
            break
          fi
      done
      conn_string="$user:$pass@(127.0.0.1:3306)/"
    fi
  fi

  if [ -z "${conn_string}" ]; then
    echo "Unable to retrieve MySQL credentials"
  else
    cat <<EOF >> "$CONFIG"
prometheus.exporter.mysql "example" {
  data_source_name = "$conn_string"
}

prometheus.scrape "mysql" {
  targets    = prometheus.exporter.mysql.example.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

EOF
    echo "MySQL integration enabled"
  fi
fi

Log Collection Config Outdated

discovery.file was renamed to local.file_match in v0.35.

Windows: Prometheus.Exporter.Windows component outdated

enabled_collectors from string to array for the prometheus.exporter.windows component needs updating from a string to an array

Logging config cannot be adjusted

The log config is set to DEBUG and cannot be changes. The log level should be configurable via environment variables.

Install script downloads the latest Grafana Agent which isn't supported properly

When using observability-agent-autoconf.sh, the service won't start properly unless I force the installer's download URL to use versions of the Grafana Agent earlier than v0.37.0 (e.g. v0.36.2).

I'm assuming there have been changes to Grafana Agent which make the installer out of date.

The Grafana Agent download URL should probably be pinned to the latest version that's been tested to work.

grafana-agent-flow.service - Vendor-neutral programmable observability pipelines.
     Loaded: loaded (/lib/systemd/system/grafana-agent-flow.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-11-22 11:34:11 GMT; 2s ago
       Docs: https://grafana.com/docs/agent/latest/flow/
    Process: 96328 ExecStart=/usr/bin/grafana-agent-flow run $CUSTOM_ARGS --storage.path=/var/lib/grafana-agent-flow $CONFIG_FILE (code=exited, status=1/FAILURE)
   Main PID: 96328 (code=exited, status=1/FAILURE)

Nov 22 11:34:11 luke-TUXEDO systemd[1]: grafana-agent-flow.service: Scheduled restart job, restart counter is at 5.
Nov 22 11:34:11 luke-TUXEDO systemd[1]: Stopped Vendor-neutral programmable observability pipelines..
Nov 22 11:34:11 luke-TUXEDO systemd[1]: grafana-agent-flow.service: Start request repeated too quickly.
Nov 22 11:34:11 luke-TUXEDO systemd[1]: grafana-agent-flow.service: Failed with result 'exit-code'.
Nov 22 11:34:11 luke-TUXEDO systemd[1]: Failed to start Vendor-neutral programmable observability pipelines..

The issue is that unix is missing a label in the created config.

Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: Error: /etc/grafana-agent-flow.river:15:1: Component "prometheus.exporter.unix" must have a label
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 14 |
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 15 | prometheus.exporter.unix {
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]:    | ^^^^^^^^^^^^^^^^^^^^^^^^
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 16 |
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: Error: /etc/grafana-agent-flow.river:20:15: component "prometheus.exporter.unix.targets" does not exist
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 19 | prometheus.scrape "unix" {
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 20 |     targets    = prometheus.exporter.unix.targets
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]:    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: 21 |     forward_to = [prometheus.remote_write.default.receiver]
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: ts=2023-12-01T13:35:51.899568476Z level=error msg="failed to start reporter" err="context canceled"
Dec 01 13:35:51 luke-TUXEDO grafana-agent-flow[54129]: Error: could not perform the initial load successfully

Windows: Elasticsearch config is incorrect

Elasticsearch config should use address not data_source_name

https://grafana.com/docs/agent/latest/flow/reference/components/prometheus.exporter.elasticsearch/

Windows: Create a no-prompt option

Look into creating the ability to run the Windows install script without it requiring input from the user.

The install script currently always prompts for additional scrape endpoints.

The Grafana Agent also pops up as a graphical installer, look into performing a silent install instead.

Add ability to configure RabbitMQ instance label via environment variable

Instance label value is currently set to one for RabbitMQ integration.

Adding the ability to configure the value via an environment variable will allow better identification.

Windows: Downloading the Grafana Agent installer is slow

Maybe it's just my VM or the powershell version but the Invoke-WebRequest, which downloads the Grafana Agent installer, is slow (takes over a minute).
Adding $ProgressPreference = 'SilentlyContinue' just before the download line speeds it up so much that it only takes a second but you can no longer see the download progress.

Needs testing on a normal Windows machine.

Windows: Grafana service complains that the river.config file isn't UTF8 encoded

Zypper install commands have the wrong syntax

Zypper manual recommends using zypper --non-interactive install ... but zypper install -y ... will do the same thing.

Windows: env variables being set instead of being compared

if ($env:variable = $true) sets the value of the env variable to true which means the conditional always evaluates to true.

Needs to be if ($env:variable -eq $true) instead. if ($env:variable) might work too but I don't know too much about powershell.

Windows - MSSQL address is incorrect

Release Process: Generate and include MD5 hash/checksum in release notes

Windows: Rabbit config doesn't use env variable correctly

address should be $env:rabbitmq_scrape_target not $rabbitmq_scrape_target