ngosang / restic-exporter Goto Github PK

Dear contributors,

you are mentioning the following in the configuration section for restic-exporter:

Configuration
[...]
All configuration is done with environment variables:
RESTIC_REPO_URL: Restic repository URL. All backends are supported
[...]
RESTIC_REPO_PASSWORD: Restic repository password in plain text. This is only required if RESTIC_REPO_PASSWORD_FILE is not defined.

I want to reuse my restic configuration file so I wrote a startup wrapper for restic-exporter sourcing the configuration and setting the value for RESTIC_REPO_PASSWORD like this:

# take all environment entries from restic backup
. /etc/dot-backup/dot-backup.conf
# set variables according to restic backup
export RESTIC_REPO_URL=${RESTIC_REPOSITORY}
export RESTIC_REPO_PASSWORD=${RESTIC_PASSWORD}
export NO_STATS=true
export NO_CHECK=true
export REFRESH_INTERVAL=3600

Unfortunately restic-exporter does not startup when executing the python file:

root@${hostname}:/usr/local/bin# restic-exporter
2023-11-07 12:21:27 INFO     Starting Restic Prometheus Exporter
2023-11-07 12:21:27 INFO     It could take a while if the repository is remote
2023-11-07 12:21:27 ERROR    The environment variable RESTIC_REPO_PASSWORD_FILE is mandatory

The code block in restic-exporter.py looks like this:

 try:
        restic_repo_url = os.environ["RESTIC_REPO_URL"]
    except Exception:
        logging.error("The environment variable RESTIC_REPO_URL is mandatory")
        sys.exit(1)

Could anybody fix this issue perhaps? I don't want to use a separate password file for this.

Many thanks in advance!
Joachim.

Related: #11 (comment)

• Add an env var to exclude the paths from the snapshot_hash
• Document the new env var and how the hash is calculated.

I was looking at the restic data in my Grafana deployment. After doing a manual backup yesterday, the client appears duplicated in the "Total backup size" item in the provided dashboard. Investigating further, I discovered that the "snapshot hash" is calculated here:

My question is, why is the snapshot hash used instead of the snapshot ID (which already is a sort of hash)?

Hi, first thanks for this great exporter.

Sometimes I get stale locks during container maintenance other activities.
I'm using this docker container for making backups from multiple machines to minio instance:

https://github.com/djmaze/resticker/blob/master/docker-compose.example.yml

...
Backup successful
Forget about old snapshots based on RESTIC_FORGET_ARGS = --prune --keep-last 10 --keep-daily 7 --keep-weekly 5 --keep-monthly 12
Fatal: unable to create lock in backend: repository is already locked by PID 839 on restic-magician by root (UID 0, GID 0)
lock was created at 2023-03-14 18:00:00 (134h1m34.969888434s ago)
storage ID 288af877

Apart from above output, stale locks can be detected by running

$ restic -q --no-lock list locks

288af8777712c23b6e4268bc20249d5a8d18c00c0b7d1ef4034dc474ba1af727

--no-lock needs to be passed because the list command creates a lock on its own.

See:

• restic/restic#1106
• https://forum.restic.net/t/detecting-stale-locks/1889
• restic/restic#2736

Then I remove them manually:

$ restic unlock

repository 6239aadd opened (version 2, compression level auto)
successfully removed 1 locks

$ restic -q --no-lock list locks

It doesn't happen too frequently so I'm not bothering modifying the docker image for backup job to add such mechanism to remove locks by default. Essentially running restic unlock before the prune command.

Still, I would like to keep track/detect those with Prometheus alert rule.
Would it be possible to expose this number via metrics?

Wrong feature request

Hello,

as you know (and support) there is a Restic REST server which also exports metrics for Prometheus. This projects metrics are considerably more detailed and useful, and it would be fantastic to have them available in the REST server. The problem problem why I'm not just using your project ootb is that this project only supports a single endpoint (right ? or did I miss something ?), and like many Restic REST servers, ours serves dozens of repositories. We'd rather not spin up dozens of copies of this service, so I was wondering whether you would be open to support copying some of your code into the REST server in general.

To be clear, I am asking permission, not for you to do the work. I know, open source and all, but considering it would be more or less copying a large part of your creative efforts, I would absolutely not feel comfortable moving forward without an okay from your side.

Unless you have a better idea, I would think translating your Python code to Go and putting it in the REST repo would probably be the most sensible solution, as their image doesn't contain any Python, and your project doesn't really lend itself to be used like a library.

Thoughts ?

I saw this error after using the Android App for Restic. It includes Restic 0.12 and the repository was created with Restic 0.13 ? => https://github.com/lhns/restic-android

ERROR    Unable to collect metrics from Restic. Error: time data '2022-12-07T10:19:24' does not match format '%Y-%m-%dT%H:%M:%S%z'

Hi,
Thanks for your work !
You're docker image, does not contain ssh for use with sftp backend.
Exemple :
docker run -d --name=restic-exporter -e TZ=Europe/Paris -e RESTIC_REPO_URL=sftp:[email protected]:/home/Nextcloud -e RESTIC_REPO_PASSWORD=yyyyyy -p 8001:8001 --restart unless-stopped ngosang/restic-exporter
Error in journalctl :
exec: "ssh": executable file not found in $PATH Exit code: 1
Alexandre

It would be nice if the script also supports variables from https://restic.readthedocs.io/en/latest/040_backup.html#environment-variables like RESTIC_REPOSITORY etc.

Then one could re-use the backup.env file for this script.

Hey,

I have a couple of restic instances backing up to different repos. I am fine with running multiple exporter instances (1 exporter per repo) but it would be nice to get an option a variable in grafan dashboard to merge all data for all/repo1/repo2 to have everything in single view :)

Hello everyone,
I want to use 'restic-exporter' with multiple repositories. Is the function already operational?
I have the same password for all these repositories, is it possible?

The goal: I am using Prometheus and Grafana on an external server, and I would like to have my repositories monitored there.

My configuration file example with multiple users: RESTIC_REPOSITORY=rest:https://login:password@url/:port/path/users/

P.S.: I can start restic-exporter with a single repository.

Thank you for your help.

Hi! Thanks for this exporter.

As explained in #8, restic-exporter calculates its own hash:

In my setup, using snapshot["paths"] as part of the input to calculate the hash causes a client to appear multiple times if the paths change, even if the rest (hostname and username) stays the same. For example, if you add or remove a path from the ones that restic is backing up for a given host and user, the exporter will report this as two different items.

I was surprised by this behavior because the metrics don't expose the paths for a given snapshot, IIRC.

Thanks for your support!

This information is not available yet.
Waiting for this feature => restic/restic#4188

Hello,

Thanks for your work. I do appreciate what you did.

I'm just asking if that's possible to get the "Paths" for printing in Grafana?

Thanks

Hi, thanks for providing this. Would it be possible to have basic auth support in and being able to set user and pass via env var?

2024-03-25 09:59:06 INFO It could take a while if the repository is remote.
2024-03-25 09:59:59 ERROR Unable to collect metrics from Restic. Error: Error executing restic snapshot command. Exit code: -9
Traceback (most recent call last):
File "/restic-exporter.py", line 226, in
prometheus_client.core.REGISTRY.register(collector)
File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 40, in register
names = self._get_names(collector)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 80, in _get_names
for metric in desc_func():
File "/restic-exporter.py", line 66, in collect
check_success.add_metric([], self.metrics["check_success"])
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'check_success'

On line 186 of restic-exporter.py you're only including the first tag in the array. So if there's multiple tags in a backup, only the first tag is showing up in Grafana.

Can line 186 be changed to:
"snapshot_tag": ",".join(snap["tags"]) if "tags" in snap else "",

to support multiple tags like how it supports multiple paths?

The only issue I can think of is if somebody wrote a dashboard or whatever and only expects there to be a single tag. If you suddenly start showing 2 or 3 tags, it may cause issues. I doubt this is an issue though but on the off chance, you could make it an optional environment variable like INCLUDE_ALLPATHS=true and default it to false. That's probably overkill though.

Hi,

maybe it's just my use case, but I found it confusing to get alerts for old snapshots which I want to keep, e.g. more than 15 days for my backups. This though happens with the example alert provided in the README file as restic-exporter reports timestamps for each snapshot, of which some might be old, yes. For my case the alert should only fire if the latest snapshot has a certain age, e.g. a backup has potentially been missed.

Maybe we like to add it to the README?

# for one day
(time() - (max without (snapshot_hash) (restic_backup_timestamp))) > (1 * 86400)

# for 15 days as currently outlined in the README
(time() - (max without (snapshot_hash) (restic_backup_timestamp))) > (15 * 86400)

Hello,

I'm trying to implement rclone with your restic-exporter, but I get an error:

restic-exporter-3 | 2023-06-14 15:27:00 ERROR Unable to collect metrics from Restic. Exception: Error executing restic snapshot command: Fatal: unable to open repository at rclone:pcloud:OVH: exec: "rclone": executable file not found in $PATH Exit code: 1

Here my docker-compose.yaml :

Is s3 supported? When setting RESTIC_REPO_URL with s3 as the http example it fails

2023-01-23 07:36:46 INFO     Starting Restic Prometheus Exporter ...
2023-01-23 07:36:46 INFO     It could take a while if the repository is remote.
Fatal: unable to open config file: Stat: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.
Is there a repository at the following location?
s3:access_key:secret_key@fqdn/bucket-name/
2023-01-23 07:36:57 ERROR    Unable to collect metrics from Restic. Error: Error executing restic snapshot command. Exit code: 1
Traceback (most recent call last):
  File "/restic-exporter.py", line 226, in <module>
    prometheus_client.core.REGISTRY.register(collector)
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 40, in register
    names = self._get_names(collector)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prometheus_client/registry.py", line 80, in _get_names
    for metric in desc_func():
  File "/restic-exporter.py", line 66, in collect
    check_success.add_metric([], self.metrics["check_success"])
                                 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'check_success'
``

Would it be possible to extract correct bytes-added and size-on-disk for the backups?
Restic does deduplication and compression and this exporter doesn't seem to take any of that into account. It appears to look at each backup separately and then sum all of that.

Here is a screenshot from Backrest making the actual backups:

While this exporter just doubles it. 🤷