mittwald / brudi

Easy, incremental and encrypted backup creation / restoration for different backends (file, mongoDB, mysql, postgres, etc.)

brudi

When it comes to backup-creation there are several solutions to use.
In general everybody's doing some sort of dump or tar and backing up the results incremental with restic or similar programs.

This is why brudi was born. brudi supports several backup-methods and is configurable by a simple yaml file. The advantage of brudi is, that you can create a backup of a source of your choice and save it with restic afterwards in one step. Under the hood, brudi uses the given binaries like mysqldump, mongodump, pg_dump, tar or restic.

Using brudi will save you from finding yourself writing bash-scripts to create your backups.

Besides creating backups, brudi can also be used to restore your data from backup in an emergency.

Table of contents

• Usage
  • CLI
  • Docker
  • Configuration
    • Sources
      • Tar
      • MySQLDump
      • MongoDump
      • PgDump
        • Limitations
      • Redis
    • Restic
      • Forget
    • Sensitive data: Environment variables
    • Gzip support for binaries without native gzip support
    • Restoring from backup
      • TarRestore
      • MongoRestore
      • MySQLRestore
      • PgRestore
        • Restore using pg_restore
        • Restore using psql
      • Restoring using restic
• Featurestate
  • Source backup methods
  • Restore backup methods
  • Incremental backup of the source backups

Usage

CLI

In order to use the brudi-binary on your local machine or a remote server of your choice, ensure you have the required tools installed.

• mongodump (required when running brudi mongodump)
• mysqldump (required when running brudi mysqldump)
• tar (required when running brudi tar)
• redis-cli (required when running brudi redisdump)
• restic (required when running brudi --restic)

$ brudi --help

Easy, incremental and encrypted backup creation for different backends (file, mongoDB, mysql, etc.)
After creating your desired tar- or dump-file, brudi backs up the result with restic - if you want to

Usage:
  brudi [command]

Available Commands:
  help           Help about any command
  mongodump      Creates a mongodump of your desired server
  mongorestore   Restores a server from a mongodump
  mysqldump      Creates a mysqldump of your desired server
  mysqlrestore   Restores a database from an sqldump
  pgdump         Creates a pg_dump of your desired postgresql-server
  pgrestore      Restores a database from a pgdump using pg_restore
  psql           Restores a database from a plain-text pgdump using psql
  redisdump      Creates an rdb dump of your desired server
  tar            Creates a tar archive of your desired 
  tarrestore     Restores files from a tar archive
  version        Print the version number of brudi

Flags:
      --cleanup         cleanup backup files afterwards
  -c, --config string   config file (default is ${HOME}/.brudi.yaml)
  -h, --help            help for brudi
      --restic          backup result with 'restic backup'
      --restic-forget   executes 'restic forget' after backing up things with restic
      --version         version for brudi

Use "brudi [command] --help" for more information about a command.

Docker

In case you don't want to install additional tools, you can also use brudi inside docker:

docker run --rm -v ${HOME}/.brudi.yml:/home/brudi/.brudi.yml quay.io/mittwald/brudi mongodump --restic --cleanup

The docker-image comes with all required binaries.

Configuration

As already mentioned, brudi is configured via .yaml. The default path for this file is ${HOME}/.brudi.yaml, but it's adjustable via -c or --config. The config file itself can include environment-variables via go-template:

restic:
  global:
    flags:
      repo: "{{ .Env.RESTIC_REPOSITORY }}"

Since the configuration provided by the .yaml-file is mapped to the corresponding CLI-flags, you can adjust literally every parameter of your source backup.
Therefore you can simply refer to the official documentation for explanations on the available flags:

• restic
• tar
• mongodump
• mysqldump
• pg_dump

Every source has a an additionalArgs-key which's value is an array of strings. The value of this key is appended to the command, generated by brudi. Even though brudi should support all cli-flags to be configured via the .yaml-file, there may be flags which are not.
In this case, use the additionalArgs-key.

It is also possible to provide more than one configuration file, for example -c mongodump.yaml -c restic.yaml. These configs get merged at runtime. If available, the default config will always be laoded first and then overwritten with any values from user-specified files. In case the same config file has been provided more than once, only the first instance will be taken into account.

Sources

Tar

tar:
  options:
    flags:
      create: true
      gzip: true
      file: /tmp/test.tar.gz
    additionalArgs: []
    paths:
      - /tmp/testfile
  hostName: autoGeneratedIfEmpty

Running: brudi tar -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
tar -c -z -f /tmp/test.tar.gz /tmp/testfile

All available flags to be set in the .yaml-configuration can be found here.

MySQLDump

mysqldump:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      opt: true
      allDatabases: true
      resultFile: /tmp/test.sqldump
    additionalArgs: []

Running: brudi mysqldump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
mysqldump --all-databases --host=127.0.0.1 --opt --password=mysqlroot --port=3306 --result-file=/tmp/test.sqldump --user=root

All available flags to be set in the .yaml-configuration can be found here.

MongoDump

mongodump:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []

Running: brudi mongodump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
mongodump --host=127.0.0.1 --port=27017 --username=root --password=mongodbroot --gzip --archive=/tmp/dump.tar.gz

All available flags to be set in the .yaml-configuration can be found here.

PgDump

pgdump:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      password: postgresroot
      username: postgresuser
      dbName: postgres
      file: /tmp/postgres.dump
    additionalArgs: []

Running: brudi pgdump -c ${HOME}/.brudi.yml --cleanup

Becomes the following command:
pg_dump --file=/tmp/postgres.dump --dbname=postgres --host=127.0.0.1 --port=5432 --username=postgresuser

All available flags to be set in the .yaml-configuration can be found here.

Limitations

Unfortunately PostgreSQL is very strict when it comes to version-compatibility.
Therefore your pg_dump-binary requires the exact same version your PostgreSQL-server is running.

The Docker-image of brudi always has the latest version available for the corresponding alpine-version installed.

Redis

redisdump:
  options:
    flags:
      host: 127.0.0.1
      password: redisdb
      rdb: /tmp/redisdump.rdb
    additionalArgs: []

Running: brudi redisdump -c ${HOME}/.brudi.yml

Becomes the following command: redis-cli -h 127.0.0.1 -a redisdb --rdb /tmp/redisdump.rdb bgsave

As redis-cli is not a dedicated backup tool but a client for redis, only a limited number of flags are available by default, as you can see here.

Restic

In case you're running your backup with the --restic-flag, you need to provide a valid configuration for restic.
You can either configure restic via brudis .yaml-configuration, or via the environment variables used by restic.

If you're already using restic in your environment, you should have everything set up perfectly to use brudi with --restic.

Forget

It's also possible to run restic forget-cmd after executing restic backup with brudi by using --restic-forget.
The forget-policy is defined in the configuration .yaml for brudi.

Example .yaml-configuration:

restic:
    global:
      flags:
        # you can provide the repository also via RESTIC_REPOSITORY
        repo: "s3:s3.eu-central-1.amazonaws.com/your.s3.bucket/myResticRepo"
    backup:
      flags:
        # in case there is no hostname given, the hostname from source backup is used
        hostname: "MyHost"
      # these paths are backuped additionally to your given source backup
      paths: []
  forget:
    flags:
      keepLast: 48
      keepHourly: 24
      keepDaily: 7
      keepWeekly: 2
      keepMonthly: 6
      keepYearly: 2
    ids: []

Sensitive data: Environment variables

In case you don't want to provide data directly in the .yaml-file, e.g. sensitive data like passwords, you can use environment-variables. Each key of the configuration is overwritable via environment-variables. Your variable must specify the whole path to a key, seperated by _.
For example, given this .yaml:

mongodump:
  options:
    flags:
      username: "" # we will override this by env
      password: "" # we will override this by env
      host: 127.0.0.1
      port: 27017
      gzip: true
      archive: /tmp/dump.tar.gz

Set your env's:

export MONGODUMP_OPTIONS_FLAGS_USERNAME="root"
export MONGODUMP_OPTIONS_FLAGS_PASSWORD="mongodbroot"

As soon as a variable for a key exists in your environment, the value of this environment-variable is used in favour of your .yaml-config.

Gzip support for binaries without native gzip support

The tools mysqldump, pg_dump and redis-cli don't natively support gzip. However, if the desired path for the backup file is suffixed with .gz, brudi will automatically gzip the backup after creation and delete the uncompressed backup file. For restoration, the file will be automatically uncompressed. Example for mysql:

mysqldump:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      opt: true
      allDatabases: true
      resultFile: /tmp/test.sqldump.gz
    additionalArgs: []

mysqlrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      Database: test
    additionalArgs: []
    sourceFile: /tmp/test.sqldump.gz

Restoring from backup

TarRestore

tarrestore:
  options:
    flags:
      extract: true
      gzip: true
      file: /tmp/test.tar.gz
      target: "/"
    additionalArgs: []
  hostName: autoGeneratedIfEmpty

Running: brudi tarrestore -c ${HOME}/.brudi.yml

Becomes the following command: tar -x -z -f /tmp/test.tar.gz -C /

MongoRestore

mongorestore:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []

Running: brudi mongorestore -c ${HOME}/.brudi.yml

Becomes the following command:
mongorestore --host=127.0.0.1 --port=27017 --username=root --password=mongodbroot --gzip --archive=/tmp/dump.tar.gz

All available flags to be set in the .yaml-configuration can be found here.

MySQLRestore

mysqlrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 3306
      password: mysqlroot
      user: root
      Database: test
    additionalArgs: []
    sourceFile: /tmp/test.sqldump

Running: brudi mysqlrestore -c ${HOME}/.brudi.yml

Becomes the following command:
mysql --database=test --host=127.0.0.1 --password=mysqlroot --port=3306 --user=root < /tmp/test.sqldump

All available flags to be set in the .yaml-configuration can be found here.

PgRestore

Restoration for PostgreSQL databases is split into two commands, psql and pgrestore. Which one to use depends on the format of the dump created with pg_dump:

psql can be used to restore plain-text dumps, which is the default format.

pgrestore can be used if the format option of pg_dump was set to tar, directory or custom.

Restore using pg_restore

pgrestore:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      username: postgresuser
      password: postgresroot
      dbname: postgres
    additionalArgs: []
    sourcefile: /tmp/postgres.dump

Running: brudi pgrestore -c ${HOME}/.brudi.yml

Becomes the following command:
pg_restore --host=127.0.0.1 --port=5432 --username=postgresuser --db-name=postgres /tmp/postgress.dump

This command has to be used if the format option was set to tar, directory or custom in pg_dump.

All available flags to be set in the .yaml-configuration can be found here.

Restore using psql

psql:
  options:
    flags:
      host: 127.0.0.1
      port: 5432
      user: postgresuser
      password: postgresroot
      dbname: postgres
    additionalArgs: []
    sourcefile: /tmp/postgres.dump

Running: brudi pgrestore -c ${HOME}/.brudi.yml

Becomes the following command: psql --host=127.0.0.1 --port=5432 --user=postgresuser --db-name=postgres < /tmp/postgress.dump

This command has to be used if the format option was set to plain in pg_dump, which is the default.

All available flags to be set in the .yaml-configuration can be found here.

Restoring using restic

Backups can be pulled from a restic repository and applied to your server by using the --restic flag in your brudi command. Example configuration for mongorestore:

mongorestore:
  options:
    flags:
      host: 127.0.0.1
      port: 27017
      username: root
      password: mongodbroot
      gzip: true
      archive: /tmp/dump.tar.gz
    additionalArgs: []
restic:
  global:
    flags:
      repo: "s3:s3.eu-central-1.amazonaws.com/your.s3.bucket/myResticRepo"
  restore:
    flags:
      target: "/"
    id: "latest"

This will pull the latest snapshot of /tmp/dump.tar.gz from the repository, which mongorestore then uses to restore the server. It is also possible to specify concrete snapshot-ids instead of latest.

Featurestate

Source backup methods

• mysqldump
• mongodump
• tar
• pg_dump
• redisdump

Restore backup methods

• mysqlrestore
• mongorestore
• tarrestore
• pgrestore
• redisrestore

Incremental backup of the source backups

• restic
  • commands
    • restic backup
    • restic forget
    • restic restore
  • storage
    • s3

brudi's Issues

Github workflow for releaseing new tags

Add release workflow

Add gzip capability for backup-binaries that don't have native`gzip` support

In order to keep the backups and save storage brudi should support gzip for backup-sources who can not gzip by default.

I'm thinking of an additional flag for brudi like brudi mysqldump --gzip which will create the mysql.dump-file and gzips it afterwards.

Another approach could be to determine whether a file should be gzipped or not by using the output-file.
If it's mysql.dump.gz, then gzip.

For now, the relevant binaries are:

• mysqldump
• redisdump
• pgsqldump

XFSdump support

Could you add xfsdump / xfsrestore support?

Create Redis backup

It would be great if Brudi was able to create backups of Redis-instances.

TBH: Since I don't know how to backup redis in a "best practice"-way, this task requires some research first.

Dump all postgres database

Hi there! Thanks for this tool!

Are there ways to backup all databases within a postgres database (e.g. calling pg_dumpall)?

Thanks!

Create integration tests using testcontainers

Write integration tests for brudi using testcontainers: https://github.com/testcontainers/testcontainers-go

Restoration of redis servers from backup

Add ability to restore a redis server from an rdb dump.

The basic workflow for this operation is the following:

1. Turn off appendonly on the server using redis-cli
2. Shutdown the server using redis-cli
3. Stop the redis service
4. Rename or remove old .aof and .rdb files for safety
5. Copy desired .rdb
6. Start the redis service

The primary issue is accomplishing 3 and 6, considering that these steps would require os-specific interactions

Fix integration-tests

The integrations-tests are executed properly on my local machine.

But there is still some magic required in order to get them work in github-actions.

Ability to restore

In case of emergency 🚨 you should be able to restore backups with brudi.

Merge multiple configs into one / make config overridable

In some cases it might be useful to give multiple configurations to brudi, similiar to helm -f values.yaml -f values.prod.yaml.

Use and parse JSON-log when executing restic instead of regex-parsing-foo

For now, we're checking what restic has done by parsing the log-output with a regex.
Since this is somehow.. weird.. we should use the json-log option from restic instead and parse the output with json.unmarshal.

https://github.com/mittwald/brudi/blob/master/pkg/restic/commands.go#L29

Implement additional restic functionality

Make already existing functions such as ListSnapshots available via flags or commands.

Change command proposal

I thinks that will be better to create commands in this pattern:

brudi <type> (dump  | restore)
Possible types: 
  mongo
  mysqldump
  pg
  psql
  redis
  tar