ngosang / restic-exporter

Prometheus exporter for the Restic backup system

ngosang/restic-exporter

Prometheus exporter for the Restic backup system.

Install

Form source code

Requirements:

• Python 3
• prometheus-client

pip install -r /requirements.txt

export RESTIC_REPOSITORY=/data
export RESTIC_PASSWORD_FILE=/restic_password_file
python restic-exporter.py

Docker

Docker images are available in GHCR and DockerHub.

docker pull ghcr.io/ngosang/restic-exporter
or
docker pull ngosang/restic-exporter

Supported Architectures

The architectures supported by this image are:

• linux/386
• linux/amd64
• linux/arm/v6
• linux/arm/v7
• linux/arm64/v8
• linux/ppc64le
• linux/s390x

docker-compose

Compatible with docker-compose v2 schemas:

---
version: '2.1'
services:
  restic-exporter:
    image: ngosang/restic-exporter
    container_name: restic-exporter
    environment:
      - TZ=Europe/Madrid
      - RESTIC_REPOSITORY=/data
      - RESTIC_PASSWORD=<password_here>
      # - RESTIC_PASSWORD_FILE=</file_with_password_here>
      - REFRESH_INTERVAL=1800 # 30 min
    volumes:
      - /host_path/restic/data:/data
    ports:
      - "8001:8001"
    restart: unless-stopped

docker cli

docker run -d \
  --name=restic-exporter \
  -e TZ=Europe/Madrid \
  -e RESTIC_REPOSITORY=/data \
  -e RESTIC_PASSWORD=<password_here> \
  -e REFRESH_INTERVAL=1800 \
  -p 8001:8001 \
  --restart unless-stopped \
  ngosang/restic-exporter

Configuration

This Prometheus exporter is compatible with all backends supported by Restic. Some of them need additional environment variables for the secrets.

All configuration is done with environment variables:

• RESTIC_REPOSITORY: Restic repository URL. All backends are supported. Examples:

  • Local repository: /data
  • REST Server: rest:http://user:[email protected]:8000/
  • Amazon S3: s3:s3.amazonaws.com/bucket_name
  • Backblaze B2: b2:bucketname:path/to/repo
  • Rclone (see notes below): rclone:gd-backup:/restic

• RESTIC_PASSWORD: Restic repository password in plain text. This is only required if RESTIC_PASSWORD_FILE is not defined.

• RESTIC_PASSWORD_FILE: File with the Restic repository password in plain text. This is only required if RESTIC_PASSWORD is not defined. Remember to mount the Docker volume with the file.

• AWS_ACCESS_KEY_ID: (Optional) Required for Amazon S3, Minio and Wasabi backends.

• AWS_SECRET_ACCESS_KEY: (Optional) Required for Amazon S3, Minio and Wasabi backends.

• B2_ACCOUNT_ID: (Optional) Required for Backblaze B2 backend.

• B2_ACCOUNT_KEY: (Optional) Required for Backblaze B2 backend.

• REFRESH_INTERVAL: (Optional) Refresh interval for the metrics in seconds. Computing the metrics is an expensive task, keep this value as high as possible. Default is 60 seconds.

• LISTEN_PORT: (Optional) The address the exporter should listen on. The default is 8001.

• LISTEN_ADDRESS: (Optional) The address the exporter should listen on. The default is to listen on all addresses.

• LOG_LEVEL: (Optional) Log level of the traces. The default is INFO.

• EXIT_ON_ERROR: (Optional) Shutdown exporter on any restic error. Default is Flase (only log error, such as network error with Cloud backends).

• NO_CHECK: (Optional) Do not perform restic check operation for performance reasons. Default is False (perform restic check).

• NO_STATS: (Optional) Do not collect per backup statistics for performance reasons. Default is False (collect per backup statistics).

• NO_LOCKS: (Optional) Do not collect the number of locks. Default is False (collect the number of locks).

• INCLUDE_PATHS: (Optional) Include snapshot paths for each backup. The paths are separated by commas. Default is False (not collect the paths).

Configuration for Rclone

Rclone is not included in the Docker image. You have to mount the Rclone executable and the Rclone configuration from the host machine. Here is an example with docker-compose:

version: '2.1'
services:
  restic-exporter:
    image: ngosang/restic-exporter
    container_name: restic-exporter
    environment:
      - TZ=Europe/Madrid
      - RESTIC_REPOSITORY=rclone:gd-backup:/restic
      - RESTIC_PASSWORD= 
      - REFRESH_INTERVAL=1800 # 30 min
    volumes:
      - /host_path/restic/data:/data
      - /usr/bin/rclone:/usr/bin/rclone:ro
      - /host_path/restic/rclone.conf:/root/.config/rclone/rclone.conf:ro
    ports:
      - "8001:8001"
    restart: unless-stopped

Exported metrics

# HELP restic_check_success Result of restic check operation in the repository
# TYPE restic_check_success gauge
restic_check_success 1.0
# HELP restic_locks_total Total number of locks in the repository
# TYPE restic_locks_total counter
restic_locks_total 1.0
# HELP restic_snapshots_total Total number of snapshots in the repository
# TYPE restic_snapshots_total counter
restic_snapshots_total 100.0
# HELP restic_backup_timestamp Timestamp of the last backup
# TYPE restic_backup_timestamp gauge
restic_backup_timestamp{client_hostname="product.example.com",client_username="root",client_version="restic 0.16.0",snapshot_hash="20795072cba0953bcdbe52e9cf9d75e5726042f5bbf2584bb2999372398ee835",snapshot_tag="mysql",snapshot_tags="mysql,tag2",snapshot_paths="/mysql/data,/mysql/config"} 1.666273638e+09
# HELP restic_backup_files_total Number of files in the backup
# TYPE restic_backup_files_total counter
restic_backup_files_total{client_hostname="product.example.com",client_username="root",client_version="restic 0.16.0",snapshot_hash="20795072cba0953bcdbe52e9cf9d75e5726042f5bbf2584bb2999372398ee835",snapshot_tag="mysql",snapshot_tags="mysql,tag2",snapshot_paths="/mysql/data,/mysql/config"} 8.0
# HELP restic_backup_size_total Total size of backup in bytes
# TYPE restic_backup_size_total counter
restic_backup_size_total{client_hostname="product.example.com",client_username="root",client_version="restic 0.16.0",snapshot_hash="20795072cba0953bcdbe52e9cf9d75e5726042f5bbf2584bb2999372398ee835",snapshot_tag="mysql",snapshot_tags="mysql,tag2",snapshot_paths="/mysql/data,/mysql/config"} 4.3309562e+07
# HELP restic_backup_snapshots_total Total number of snapshots
# TYPE restic_backup_snapshots_total counter
restic_backup_snapshots_total{client_hostname="product.example.com",client_username="root",client_version="restic 0.16.0",snapshot_hash="20795072cba0953bcdbe52e9cf9d75e5726042f5bbf2584bb2999372398ee835",snapshot_tag="mysql",snapshot_tags="mysql,tag2",snapshot_paths="/mysql/data,/mysql/config"} 1.0
# HELP restic_scrape_duration_seconds Amount of time each scrape takes
# TYPE restic_scrape_duration_seconds gauge
restic_scrape_duration_seconds 166.9411084651947

Prometheus config

Example Prometheus configuration:

scrape_configs:
  - job_name: 'restic-exporter'
    static_configs:
      - targets: ['192.168.1.100:8001']

Prometheus / Alertmanager rules

Example Prometheus rules for alerting:

  - alert: ResticCheckFailed
    expr: restic_check_success == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Restic check failed (instance {{ $labels.instance }})
      description: Restic check failed\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}

  - alert: ResticOutdatedBackup
    # 1209600 = 15 days
    expr: time() - restic_backup_timestamp > 1209600
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: Restic {{ $labels.client_hostname }} / {{ $labels.client_username }} backup is outdated
      description: Restic backup is outdated\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}

Grafana dashboard

There is a reference Grafana dashboard in grafana/grafana_dashboard.json.

restic-exporter's Issues

multi repository

Hello everyone,
I want to use 'restic-exporter' with multiple repositories. Is the function already operational?
I have the same password for all these repositories, is it possible?

The goal: I am using Prometheus and Grafana on an external server, and I would like to have my repositories monitored there.

My configuration file example with multiple users: RESTIC_REPOSITORY=rest:https://login:password@url/:port/path/users/

P.S.: I can start restic-exporter with a single repository.

Thank you for your help.

[Question] Why is snapshot_hash used instead of snapshot ID?

I was looking at the restic data in my Grafana deployment. After doing a manual backup yesterday, the client appears duplicated in the "Total backup size" item in the provided dashboard. Investigating further, I discovered that the "snapshot hash" is calculated here:

restic-exporter/restic-exporter.py

Line 201 in 880b471

def calc_snapshot_hash(self, snapshot: dict) -> str:

My question is, why is the snapshot hash used instead of the snapshot ID (which already is a sort of hash)?

docker image ssh for sftp backend

Hi,
Thanks for your work !
You're docker image, does not contain ssh for use with sftp backend.
Exemple :
docker run -d --name=restic-exporter -e TZ=Europe/Paris -e RESTIC_REPO_URL=sftp:[email protected]:/home/Nextcloud -e RESTIC_REPO_PASSWORD=yyyyyy -p 8001:8001 --restart unless-stopped ngosang/restic-exporter
Error in journalctl :
exec: "ssh": executable file not found in $PATH Exit code: 1
Alexandre

[FEATURE] Multiple repo support

Hey,

I have a couple of restic instances backing up to different repos. I am fine with running multiple exporter instances (1 exporter per repo) but it would be nice to get an option a variable in grafan dashboard to merge all data for all/repo1/repo2 to have everything in single view :)

Error: time data '2022-12-07T10:19:24' does not match format '%Y-%m-%dT%H:%M:%S%z'

I saw this error after using the Android App for Restic. It includes Restic 0.12 and the repository was created with Restic 0.13 ? => https://github.com/lhns/restic-android

ERROR    Unable to collect metrics from Restic. Error: time data '2022-12-07T10:19:24' does not match format '%Y-%m-%dT%H:%M:%S%z'

Exclude paths from snapshot_hash

Related: #11 (comment)

• Add an env var to exclude the paths from the snapshot_hash
• Document the new env var and how the hash is calculated.

Environment variable "RESTIC_REPO_PASSWORD" is not read by restic-exporter.py

Dear contributors,

you are mentioning the following in the configuration section for restic-exporter:

Configuration
[...]
All configuration is done with environment variables:
RESTIC_REPO_URL: Restic repository URL. All backends are supported
[...]
RESTIC_REPO_PASSWORD: Restic repository password in plain text. This is only required if RESTIC_REPO_PASSWORD_FILE is not defined.

I want to reuse my restic configuration file so I wrote a startup wrapper for restic-exporter sourcing the configuration and setting the value for RESTIC_REPO_PASSWORD like this:

# take all environment entries from restic backup
. /etc/dot-backup/dot-backup.conf
# set variables according to restic backup
export RESTIC_REPO_URL=${RESTIC_REPOSITORY}
export RESTIC_REPO_PASSWORD=${RESTIC_PASSWORD}
export NO_STATS=true
export NO_CHECK=true
export REFRESH_INTERVAL=3600

Unfortunately restic-exporter does not startup when executing the python file:

root@${hostname}:/usr/local/bin# restic-exporter
2023-11-07 12:21:27 INFO     Starting Restic Prometheus Exporter
2023-11-07 12:21:27 INFO     It could take a while if the repository is remote
2023-11-07 12:21:27 ERROR    The environment variable RESTIC_REPO_PASSWORD_FILE is mandatory

The code block in restic-exporter.py looks like this:

 try:
        restic_repo_url = os.environ["RESTIC_REPO_URL"]
    except Exception:
        logging.error("The environment variable RESTIC_REPO_URL is mandatory")
        sys.exit(1)

Could anybody fix this issue perhaps? I don't want to use a separate password file for this.

Many thanks in advance!
Joachim.

rclone is available with restic-exporter?

Hello,

I'm trying to implement rclone with your restic-exporter, but I get an error:

restic-exporter-3 | 2023-06-14 15:27:00 ERROR Unable to collect metrics from Restic. Exception: Error executing restic snapshot command: Fatal: unable to open repository at rclone:pcloud:OVH: exec: "rclone": executable file not found in $PATH Exit code: 1

Here my docker-compose.yaml :

Collaboration with Restic-REST backend ?

Hello,

as you know (and support) there is a Restic REST server which also exports metrics for Prometheus. This projects metrics are considerably more detailed and useful, and it would be fantastic to have them available in the REST server. The problem problem why I'm not just using your project ootb is that this project only supports a single endpoint (right ? or did I miss something ?), and like many Restic REST servers, ours serves dozens of repositories. We'd rather not spin up dozens of copies of this service, so I was wondering whether you would be open to support copying some of your code into the REST server in general.

To be clear, I am asking permission, not for you to do the work. I know, open source and all, but considering it would be more or less copying a large part of your creative efforts, I would absolutely not feel comfortable moving forward without an okay from your side.

Unless you have a better idea, I would think translating your Python code to Go and putting it in the REST repo would probably be the most sensible solution, as their image doesn't contain any Python, and your project doesn't really lend itself to be used like a library.

Thoughts ?

Basic Auth support

Hi, thanks for providing this. Would it be possible to have basic auth support in and being able to set user and pass via env var?

Add support for reporting number of (stale) locks

Hi, first thanks for this great exporter.

Sometimes I get stale locks during container maintenance other activities.
I'm using this docker container for making backups from multiple machines to minio instance:

https://github.com/djmaze/resticker/blob/master/docker-compose.example.yml

...
Backup successful
Forget about old snapshots based on RESTIC_FORGET_ARGS = --prune --keep-last 10 --keep-daily 7 --keep-weekly 5 --keep-monthly 12
Fatal: unable to create lock in backend: repository is already locked by PID 839 on restic-magician by root (UID 0, GID 0)
lock was created at 2023-03-14 18:00:00 (134h1m34.969888434s ago)
storage ID 288af877

Apart from above output, stale locks can be detected by running

$ restic -q --no-lock list locks

288af8777712c23b6e4268bc20249d5a8d18c00c0b7d1ef4034dc474ba1af727

--no-lock needs to be passed because the list command creates a lock on its own.

See:

• restic/restic#1106
• https://forum.restic.net/t/detecting-stale-locks/1889
• restic/restic#2736

Then I remove them manually:

$ restic unlock

repository 6239aadd opened (version 2, compression level auto)
successfully removed 1 locks

$ restic -q --no-lock list locks

It doesn't happen too frequently so I'm not bothering modifying the docker image for backup job to add such mechanism to remove locks by default. Essentially running restic unlock before the prune command.

Still, I would like to keep track/detect those with Prometheus alert rule.
Would it be possible to expose this number via metrics?

Alert example fires for maintained snapshots

Hi,

maybe it's just my use case, but I found it confusing to get alerts for old snapshots which I want to keep, e.g. more than 15 days for my backups. This though happens with the example alert provided in the README file as restic-exporter reports timestamps for each snapshot, of which some might be old, yes. For my case the alert should only fire if the latest snapshot has a certain age, e.g. a backup has potentially been missed.

Maybe we like to add it to the README?

# for one day
(time() - (max without (snapshot_hash) (restic_backup_timestamp))) > (1 * 86400)

# for 15 days as currently outlined in the README
(time() - (max without (snapshot_hash) (restic_backup_timestamp))) > (15 * 86400)

Invalid: feature request

Wrong feature request

Support for multiple tags?

On line 186 of restic-exporter.py you're only including the first tag in the array. So if there's multiple tags in a backup, only the first tag is showing up in Grafana.

Can line 186 be changed to:
"snapshot_tag": ",".join(snap["tags"]) if "tags" in snap else "",

to support multiple tags like how it supports multiple paths?

The only issue I can think of is if somebody wrote a dashboard or whatever and only expects there to be a single tag. If you suddenly start showing 2 or 3 tags, it may cause issues. I doubt this is an issue though but on the off chance, you could make it an optional environment variable like INCLUDE_ALLPATHS=true and default it to false. That's probably overkill though.

Duplicated snapshots because of paths

Hi! Thanks for this exporter.

As explained in #8, restic-exporter calculates its own hash:

restic-exporter/restic-exporter.py

Lines 286 to 288 in a4e37f8

	def calc_snapshot_hash(self, snapshot: dict) -> str:
	text = snapshot["hostname"] + snapshot["username"] + ",".join(snapshot["paths"])
	return hashlib.sha256(text.encode("utf-8")).hexdigest()

In my setup, using snapshot["paths"] as part of the input to calculate the hash causes a client to appear multiple times if the paths change, even if the rest (hostname and username) stays the same. For example, if you add or remove a path from the ones that restic is backing up for a given host and user, the exporter will report this as two different items.

I was surprised by this behavior because the metrics don't expose the paths for a given snapshot, IIRC.

Thanks for your support!

Possible to get the "Paths"

Hello,

Thanks for your work. I do appreciate what you did.

I'm just asking if that's possible to get the "Paths" for printing in Grafana?

Thanks

S3 Support?

Is s3 supported? When setting RESTIC_REPO_URL with s3 as the http example it fails

2023-01-23 07:36:46 INFO     Starting Restic Prometheus Exporter ...
2023-01-23 07:36:46 INFO     It could take a while if the repository is remote.
Fatal: unable to open config file: Stat: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
Is there a repository at the following location?
s3:access_key:secret_key@fqdn/bucket-name/
2023-01-23 07:36:57 ERROR    Unable to collect metrics from Restic. Error: Error executing restic snapshot command. Exit code: 1
Traceback (most recent call last):
  File "/restic-exporter.py", line 226, in <module>
    prometheus_client.core.REGISTRY.register(collector)
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 40, in register
    names = self._get_names(collector)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 80, in _get_names
    for metric in desc_func():
  File "/restic-exporter.py", line 66, in collect
    check_success.add_metric([], self.metrics["check_success"])
                                 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'check_success'
``

Include Restic client version metric

This information is not available yet.
Waiting for this feature => restic/restic#4188

Also allow restic environment variables

It would be nice if the script also supports variables from https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables like RESTIC_REPOSITORY etc.

Then one could re-use the backup.env file for this script.