alphapapa / restic-runner

Configure and run Restic more easily

restic-runner

restic-runner is a helper script for using restic. It makes it easy to configure backup repositories and data sets, and to operate on them.

Contents

• Installation
• Configuration
• Usage
• Tips

Installation

1. Put the restic-runner script in your PATH.
2. Make the directory ~/.config/backup/restic (or a path of your choosing after changing the config_dir default variable in the script).

MacOS

On MacOS, you will probably need to install GNU getopt, as well as GNU date (symlinking it to date as needed). You may want to put them earlier in your PATH when running restic-runner. For example, using Homebrew, you could do this:

$ brew install gnu-getopt coreutils
$ ln -s /usr/local/bin/gdate $HOME/.bin/date
$ export PATH="/usr/local/opt/gnu-getopt/bin:$HOME/.bin:$PATH"

Note: The author does not use MacOS, so he cannot directly vouch for this information. See #5, #7, #8, #9.

Additional note: This project is designed for use on free, open platforms such as GNU/Linux, BSDs, etc. Users of proprietary platforms (especially ones like MacOS, which bundle outdated and non-standard tools) may have to make various minor modifications for compatibility. Such issues are not considered bugs in this project.

Configuration

All configuration files are actually Bash scripts, which are sourced by restic-runner. Therefore, the following configuration instructions are actually just a convention. That means that any of the variables mentioned may actually be specified in either repository or set configuration files, or both (e.g. when using Bash arrays, both files could add to an array by using array_var+=(element1 element2) syntax). Obviously, if a variable is set in both files (as opposed to being appended to), the resulting behavior should be considered undefined (i.e. don’t rely on the order in which the files are sourced; keep it simple).

Global

A global configuration file may be put at ~/.config/backup/restic/runner. You may use it to set any variables that you might also set in the following files. This file will be sourced first, so repo- and set-specific files will override settings in this file.

This is especially useful for settings that are Bash arrays, like exclude_patterns, because you can use += syntax to add to the array in each file (or you could use regular assignment to override rather than add to).

Repositories

To configure a backup repository, put a file at ~/.config/backup/restic/repos/REPONAME. It should export these variables:

• RESTIC_REPOSITORY: The path to the repository root
• RESTIC_PASSWORD_FILE (path to a file containing the plain-text password for the repo) or RESTIC_PASSWORD (the password for the repo in plain-text)

It may also define these variables:

• du: Set to true to report the repo size and difference after running a Restic command. This should only be enabled for local repos (e.g. not SFTP ones).
• keep_policy: A Bash array containing a list of --keep-X options.

Example:

export RESTIC_REPOSITORY=/mnt/backup/restic/REPONAME
export RESTIC_PASSWORD_FILE=$RESTIC_REPOSITORY/password_file

du=true

keep_policy=(
    --keep-last 2
    --keep-daily 7
    --keep-weekly 8
    --keep-monthly 12
    --keep-yearly 10
)

Data sets

To configure a backup data set, put a file at ~/.config/backup/restic/sets/SETNAME. It should define these variables:

• tag: The tag to apply to the data set in Restic.
• include_paths: A Bash array of paths to include in the backup. Unquoted paths will be expanded by Bash, so you can use ~ for your home directory. Remember to quote paths (or parts of paths) containing spaces! Unquoted globs will also be expanded by the shell, which you almost certainly never want, so you should always quote glob patterns.
• exclude_patterns: A Bash array of patterns to exclude from the backup. The same rules about quoting and shell expansion apply: paths can be unquoted (except for ones containing spaces), and patterns should be quoted.
• exclude_if_present: A Bash array of filenames to pass to Restic using the --exclude-if-present option.

Example:

tag=main

include_paths=(
    # This comment is ignored by Bash, so you can comment your configuration freely.
    ~/src
    ~/"Important Files"  # Quote the part containing a space, but leave ~ unquoted so Bash will expand it
)

exclude_if_present+=(
    # Using += instead of = so this can also be set by repo config files.
    .nobackup
    .resticignore
)

exclude_patterns=(
    # Backup files
    "*~"
    "*.bak"
    "/**/.backup"

    # Temp files
    "/#*#"
    "/.#*"

    # Misc
    ~/tmp
)

Usage

Run restic-runner with these options:

• --repo REPONAME: Use the name of the repo file you configured.
• --set SETNAME: Use the name of the data set file you configured.

After each command, a log is displayed giving the duration, current repo size, and change in repo size:

LOG (2018-01-06 00:09:57-06:00): backup finished.  Duration: 7m31s  Repo size: 50.341 GB (+1.576 GB)

Combined with the diff command and the --added filter, this makes it easy to find out why your repo suddenly increased in size.

Commands

• backup
• check
• command COMMAND-STRING
• diff {SNAPSHOT1} {SNAPSHOT2}
• expire
• init
• mount PATH
• snapshot-ids
• verify-randomly {N}

backup

Runs a backup. By default it calls Restic with these options:

• --one-file-system
• --exclude-caches

Example:

restic-runner --repo REPONAME --set SETNAME backup

check

Runs restic check on the repo. The --set option may be omitted, since it is meaningless for this command.

Example:

restic-runner --repo REPONAME check

command COMMAND-STRING

Pass the specified command through to restic. This is useful for commands that do not have an equivalent in restic-runner.

Example:

# Forget specific snapshots
restic-runner --repo REPONAME command forget abcd1234 deadbeef

Note: To avoid further processing of options that are valid for both restic and restic-runner, use --, like:

# Prevent "--tag main" from being processed by restic-runner; instead, pass it to restic
restic-runner --repo REPONAME -- command snapshots --tag main

Which results in running restic snapshots --tag main.

If unsure, you can use restic-runner --debug to see how arguments are parsed.

diff [SNAPSHOT1] [SNAPSHOT2]

Shows the diff between two snapshots. One or more snapshot IDs may optionally be specified, oldest to newest:

• If none are given, the latest snapshot is compared with the one before it.
• If one is given, it is compared with the latest.
• If two are given, they are compared.

Each snapshot ID may be suffixed by one or more ^ to refer to the snapshot that-many snapshots before it. HEAD refers to the latest snapshot.

These options may be specified (before the command):

• --added Show only added paths
• --modified Show only modified paths
• --removed Show only removed paths

--added and --modified may be used together.

Examples:

# Show the paths added in the latest snapshot.
restic-runner --repo REPONAME --added diff HEAD^

# Show the paths added and modified between snapshot ID "deadbeef" and
# the snapshot before it.
restic-runner --repo REPONAME --added --modified diff deadbeef^ deadbeef

expire

Automatically forget and prune snapshots according to the configured policy.

Example:

restic-runner --repo REPONAME expire

forget

Forget snapshots according to the configured policy.

Example:

restic-runner --repo REPONAME --set SETNAME forget

init

Initialize the configured repo.

Example:

restic-runner --repo REPONAME init

mount PATH

Mount the repo to PATH.

Example:

restic-runner --repo REPONAME mount ~/mnt/restic

prune

Prune snapshots in the repo.

Example:

restic-runner --repo REPONAME prune

snapshot-ids

Print a list of snapshot IDs, one per line.

Example:

# Print all snapshot IDs for the repo
restic-runner --repo REPONAME snapshot-ids

# Print snapshot IDs for the tag configured in this set
restic-runner --repo REPONAME --set SETNAME snapshot-ids

# Print snapshot IDs for this tag
restic-runner --repo REPONAME --tag TAG snapshot-ids

verify-randomly [N]

Verify N (default 10) random files from the latest snapshot. If --compare is specified, the restored files are compared with the live versions.

Note that the --set SETNAME option may be specified to e.g. choose the latest snapshot in SETNAME, or omitted to e.g. choose the latest snapshot in the repo.

These options may be specified:

• --compare Compare restored files with live versions, exiting with an error if any differ.
• --snapshot SNAPSHOT-ID Restore from this snapshot.

Examples:

# Verify 10 random files from the latest snapshot in set SETNAME.
restic-runner --repo REPONAME --set SETNAME verify-randomly

# Verify and compare 100 random files from snapshot DEADBEEF with verbose output.
restic-runner -v --repo REPONAME --snapshot deadbeef --compare verify-randomly 100

Tips

• When running in a cron job, use the chronic utility from moreutils, which only sends output if the job exits with non-zero status. (However, this means you’ll only receive the log if an error occurs, so it won’t be as easy to notice if your repo suddenly grows due to unintentionally backing up some files.)
• Repo and set config files can be placed in subdirectories of their respective directories. For example, the repo config file ~/.config/backup/restic/repos/remote/s3 can be referred to like restic-runner --repo remote/s3.

License

GPLv3

restic-runner's Issues

How do I pass this parameter -o b2.connections=50 to the restic-runner script

When backing up to backblaze b2 , restic uses the default value of -o b2.connections=5 (https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html ) which makes it pretty slow.It's been recommended to us b2.connections=50 (restic/restic#1383)

e.g. restic -r b2:<bucket_name>:<b2_folder> -o b2.connections=50 backup ~/test_folder

When I try to put this value in the REPONAME file, it throws an error
export RESTIC_REPOSITORY=b2:<bucket_name>:<b2_folder> -o b2.connections=50

I tried quoting it too but doesn't work.
Is there a way I can pass this value to the restic-runner script?

Wrong exit status if no error occurred

If the script runs without errors, it returns an exit status of 1.

Since the errors variable is only initialized in the error() function, that is never called when there are no errors encountered, it is undefined and evaluates to '' in the final test before the script exits.
Therefore the test:

restic-runner/restic-runner

Line 606 in e592a4d

[[ $errors -gt 0 ]] && log "Errors: $errors"

evaluates to 1 and bash uses that value to exit the script.

Initializing errors at the start of the script fixes the issue.

Edit: Nevermind, I just saw the PR adressing the issue: #20

./restic-runner: line 91: fusermount: command not found

I encounter this error

./restic-runner: line 91: fusermount: command not found

while running

restic-runner --repo REPONAME --set SETNAME verify-randomly

I am on macOS Catalina 10.15.7 (19H2) with GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.5.0)

Fixed this by replacing Line 91 with

umount "$mount_point" || error "Unable to unmount repo from: $mount_point"

Adding variable for restic binary

Hello,

To solve some crontab problems, I added the variable restic_bin at the beginning of the script and replaced the calls to restic by $restic_bin.

[RFE] use keep_policy per data set instead of repo

Currently the keep_policy is defined once per repository. In some cases one might want to apply different policies per data set.

Allow specifying hostname when backing up

Would be nice to be able to configure the --host argument to the backup command.

My shell (zsh on macos) has a HOST environment variable, so an easy fix for me is to simply add --host "$HOST" \ to the backup command, and then exporting HOST in the runner config file. This is probably not portable though.

tempfile: command not found

/usr/local/bin/restic-runner: line 73: tempfile: command not found

If I understand correctly the script tries to create a temporary file out of the exclude_file, but the corresponding functions is missing.

syntax error: operand expected

OS: CentOS Linux release 7.4.1708 (Core)

/usr/local/bin/restic-runner: line 348: -  : syntax error: operand expected (error token is "-  ")
(standard_in) 1: syntax error
/usr/local/bin/restic-runner: line 350: - : syntax error: operand expected (error token is "- ")
date: invalid date ‘Sat  6 Jan 14:57:54 CET 2018’
date: invalid date ‘Sat  6 Jan 15:07:12 CET 2018’
/usr/local/bin/restic-runner: line 124: -  : syntax error: operand expected (error token is "-  ")

Please note that due to #1 I was using an older commit, so lines don't match the current master version.

./restic-runner: line 414: readarray: command not found ERROR: Unable to list paths in snapshot

I encounter this error

./restic-runner: line 414: readarray: command not found ERROR: Unable to list paths in snapshot: <snapshotid>

while running

restic-runner --repo REPONAME --set SETNAME verify-randomly

I am on macOS Catalina 10.15.7 (19H2) with GNU bash, version 5.0.18(1)-release (x86_64-apple-darwin19.5.0)

Issue running on macOS

hi all, when I'm trying to run the latest version on macOS Mojave I get stuck in the following loop:

restic-runner -h
LOG (2019-03-02 13:20:04): REPO: SET: COMMAND:/Users/eim/bin/restic-runner
LOG (2019-03-02 13:20:04): REPO: SET: COMMAND:/Users/eim/bin/restic-runner
LOG (2019-03-02 13:20:04): REPO: SET: COMMAND:/Users/eim/bin/restic-runner
...

the same script works fine on Linux. Some infos:

~ zsh --version
zsh 5.3 (x86_64-apple-darwin18.0)
~ bash --version
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
Copyright (C) 2007 Free Software Foundation, Inc.

when I comment

        --)
            # Remaining args
            shift
            command="$1"
            shift
            rest=("$@")
            break
            ;;

at least I come out of the loop, any sugggestions? Thanks

snapshot-ids does not work

Here is the debug output

./restic-runner --debug --repo abcxyz snapshot-ids

DEBUG: ARGS: --debug --repo 'abcxyz' -- 'snapshot-ids'
DEBUG: Command: snapshot-ids
DEBUG: Rest:
LOG (2019-03-24 17:43:34): REPO:abcxyz SET: COMMAND:snapshot-ids
LOG (2019-03-24 17:44:00): snapshot-ids FINISHED. Duration: 26s

/usr/bin/restic-runner: line 242: - : syntax error: operand expected (error token is "- ")

I'm running Fedora 36 and am trying to run a backup.

I am getting this error.

date: invalid date ‘Thu Aug 11 10:30:36 PM -03 2022’
date: invalid date ‘Thu Aug 11 10:30:50 PM -03 2022’
/usr/bin/restic-runner: line 242: -  : syntax error: operand expected (error token is "-  ")

The full output is as follows.

[i@fedora ~]$ restic-runner --repo amd1 --set sony backup
LOG (2022-08-11 22:30:36): REPO:amd1 SET:sony COMMAND:backup
LOG (2022-08-11 22:30:36): BACKING UP...
repository 2ae84258 opened successfully, password is correct
using parent snapshot 318149f9

Files:           0 new,     0 changed, 15589 unmodified
Dirs:            0 new,     0 changed,  2391 unmodified
Added to the repo: 0 B  

processed 15589 files, 8.556 GiB in 0:08
snapshot 625ad82b saved
date: invalid date ‘Thu Aug 11 10:30:36 PM -03 2022’
date: invalid date ‘Thu Aug 11 10:30:50 PM -03 2022’
/usr/bin/restic-runner: line 242: -  : syntax error: operand expected (error token is "-  ")
LOG (2022-08-11 22:30:50): backup FINISHED.  Duration:  

It seems to have worked just fine though. Except for the errors being shown and the duration being blank.

Documentation: why password is stored on same repo folder

On project's README, the config file sample for a repo has the following lines:

export RESTIC_REPOSITORY=/mnt/backup/restic/REPONAME
export RESTIC_PASSWORD_FILE=$RESTIC_REPOSITORY/password_file

If I get it right, on your example, the password for a repo is stored in a file right next to the repo itself. I understand that this might be just a matter of preference, but maybe there's also some rationale behind it?
I'd guess it makes the backup more "portable", in the sense that if you have the backup drive with both the restic repo and the password for such repo, then you have everything you need to access/restore data from repo. On the other hand, it seems to make the backup location less secure, as an attacker getting access to the backup location can gets his hands both on the backup itself and on the password to access the data on backup.

PS: I'm still wrapping my head around restic and restic-runner.

snapshot-ids does not work as described in the readme (and fix)

In the readme it is described that the command

restic-runner --repo REPONAME snapshot-ids

should list the snapshots stored in the repositry.

The output of the command on my installation shows no list of the snapshot ids in the repository:

$ restic-runner -v --debug --repo restic-repo snapshot-ids
DEBUG: ARGS:  -v --debug --repo 'restic-repo' -- 'snapshot-ids'
DEBUG: Command: snapshot-ids
DEBUG: Rest: 
LOG (2023-02-19 16:57:44): REPO:restic-repo SET: COMMAND:snapshot-ids
LOG (2023-02-19 16:57:45): snapshot-ids FINISHED.  Duration: 1s  

Passing the tag, it works properly, although filtering by tag does not list all the snapshots in the repository:

$ restic-runner -v --debug --repo restic-repo --set restic-repo-set snapshot-ids
DEBUG: ARGS:  -v --debug --repo 'restic-repo' --set 'restic-repo-set' -- 'snapshot-ids'
DEBUG: Command: snapshot-ids
DEBUG: Rest: 
LOG (2023-02-19 16:57:48): REPO:restic-repo SET:restic-repo-set COMMAND:snapshot-ids
ce5b515c
4c72a26f
38307900
...
...
LOG (2023-02-19 16:57:49): snapshot-ids FINISHED.  Duration: 1s  

I think there is a small typo in the snapshot-ids function:

    restic snapshots \
           --tag "$tag" \
           ...

should be

    restic snapshots \.
           $(tag) \
           ...

Then:

$ restic-runner -v --debug --repo restic-repo snapshot-ids
DEBUG: ARGS:  -v --debug --repo 'restic-repo' -- 'snapshot-ids'
DEBUG: Command: snapshot-ids
DEBUG: Rest: 
LOG (2023-02-19 17:35:28): REPO:restic-repo SET: COMMAND:snapshot-ids
ac05df29
ce5b515c
4c72a26f
38307900
...
LOG (2023-02-19 17:35:29): snapshot-ids FINISHED.  Duration: 0s 

I don't know if this will cause problems with the other commands, but so far I haven't noticed anything unusual.

Bye!

snapshot-ids does not work

This is the output I get when I run snapshot-ids.

./restic-runner --repo abcxyz snapshot-ids

LOG (2019-03-24 00:21:26): REPO:abcxyz SET: COMMAND:snapshot-ids
LOG (2019-03-24 00:21:58): snapshot-ids FINISHED. Duration: 32s

du: cannot access

du: cannot access ‘sftp:restic-server:backup’: No such file or directory

Obviously du doesn't work on SFTP repositories. I think you already considered to make du optional. Just wanted to add this issue as a reminder.

Accumulating temporary files

For every backup run, the script creates a new temporary file with the specified exclude patterns as its content.

I think the exclude_file function is executed in a subshell and the exclude_file variable goes out of scope, thus the cleanup function actually never gets a valid path to the temporary file.

restic-runner/restic-runner

Line 253 in 10dbc57

--exclude-file $(exclude_file) \

Modifying cleanup to accept the file path as an argument from the trap in backup works as expected.

Local metadata cache dir parameter in global

Hello,

Backup of ~250GB of datasets (photos) results in a 30 minutes run, even though nothing changed.

Reading through it, it seems there's a metadata cache feature:
restic/restic#1040

I would like to activate it, is it possible to set it up in the "global parameters" file? if yes, how?

Thanks!

Complete set of instructions for MacOS. Please add to README

Here are the complete list of instructions I did to fix all issues on MacOS .Please add them to the README

#Install the following applications using Homebrew. Follow linking instructions for each app as it gets installed
brew install restic rclone moreutils gnu-getopt coreutils bash
#Add new export Paths to ~/.bash_profile
echo 'export PATH="/usr/local/opt/gnu-getopt/bin:/usr/local/opt/coreutils/libexec/gnubin:$PATH"' >> ~/.bash_profile
#Change Shell
sudo bash -c 'echo /usr/local/bin/bash >> /etc/shells' && chsh -s /usr/local/bin/bash

#Replace the shebang in the restic-runner script
echo '#!/usr/bin/env bash' | cat - restic-runner > temp && mv temp restic-runner

#Set the value of config_dir
config_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"/config

forget policy variables to command don't work

forget policy variables to command don't work e.g.

If I run this

restic-runner --repo REPONAME command forget --keep-last 5 --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 5

or

restic-runner --repo REPONAME command "forget --keep-last 5 --keep-daily 7 --keep-weekly 5 --keep-monthly 12 --keep-yearly 5"

I get this error

restic-runner: unrecognized option `--keep-last'
restic-runner: unrecognized option `--keep-daily'
restic-runner: unrecognized option `--keep-weekly'
restic-runner: unrecognized option `--keep-monthly'
restic-runner: unrecognized option `--keep-yearly'

date: illegal time format on MacOS Mojave

I get this error when I run in on MacOS Mojave

restic-runner --repo REPONAME init

LOG (2019-03-16 01:55:48): REPO:REPONAME SET: COMMAND:init
LOG (2019-03-16 01:55:48): INITIALIZING...
created restic repository 26b298bd7f at rclone:onedrive:backups

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
restic-runner: line 187: - : syntax error: operand expected (error token is " ")