sierrasoftworks / minback-cleanup

A backup rotation tool for Minio backups created by the various minback containers

minback-cleanup

Minio Rolling Backup Management Container

This container provides a clean, job based, approach to rotating backups on a schedule according to given rules. It is intended to be used in conjunction with the various minback containers which place backup files in a Minio bucket for you.

Features

• Allows multiple backup retention periods to be selected
• Lightweight and short lived
• Simple and well tested Go implementation

Example

In this example, let's assume we have a very aggressive backup schedule that creates a new backup every minute. We would like to keep:

• Every backup for the last 1h
• A backup every 15m after 1h
• A backup every 1h after 6h
• A backup every week (7d) after 1 week
• A backup every 4 weeks (4w) after a month (4w)
• A backup every year after 3 years (156w)

docker run --rm --env-file backup.env minback/cleanup cleanup --db my_db --keep "@1h/15m~5m" --keep "@6h/1h~15m" --keep "@7d/1w~1d" --keep "@4w/4w~1w" --keep "@52w/156w~1w"

backup.env

MINIO_SERVER=https://play.minio.io
MINIO_ACCESS_KEY=minio
MINIO_SECRET_KEY=miniosecret
MINIO_BUCKET=backups

Keep Specifications

A keep specification is composed of three different properties:

1. Window (@) tells us when this window starts to apply from. It defaults to "right now" and can be extended out with a time duration, for example 52w tells us that the window starts at 1 year and continues into the past from there.

   Windows "stack", so @52w and @4w will combine to have one specification applying from 4w to 52w and the next from 52w onwards.

2. Spacing (/) tells us which backups within a window to keep. A spacing of 15m will keep any backups which fall on the 15m boundary (according to their timestamp) - so :00, :15, :30 and :45.

3. Smudge Factor (~) tells us that we should round the backup's time to the nearest provided duration. This is great for situations where your backup's timestamp is a lot more accurate than your window and you wish to smudge things to ensure it falls on a boundary.

   For example, a smudge of ~15m will convert 11:13:18 to 11:15:00 and have it match the /15m spacing we selected previously. You can also use larger smudges like ~1d or even ~52w (for a year), but keep in mind that we don't de-duplicate things if multiple backups match because of the same smudge factor.

You define a keep specification by concatenating each property's control character with its duration and listing them in any order you wish.

Example: @2w/1d~6h

In this example, let's assume we create a new backup every 6 hours but only want to store one backup per day for everything older than 2 weeks (i.e. your 6 hour frequency is for short-term recovery).

1. Window is 2 weeks (2w), meaning this specification will only apply to backups older than 2 weeks, and until an older window is found.
2. Spacing is 1 day (1d), meaning we will keep one backup per day within this window. Backups will be selected based on their timestamp and only those which match (after smudging) the day boundary will be kept.
3. Smudge Factor is 6 hours (6h) which ensures that backups which occur up to 3 hours either side of the day boundary (midnight) will be kept.

Usage

NAME:
   main.exe - A new cli application

USAGE:
   main.exe [global options] command [command options] [arguments...]

VERSION:
   v1.0.0

AUTHOR:
   Benjamin Pannell <[email protected]>

COMMANDS:
     cleanup
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --log-level value  DEBUG|INFO|WARN|ERROR (default: "INFO")
   --help, -h         show help
   --version, -v      print the version

COPYRIGHT:
   Sierra Softworks © 2017

cleanup

NAME:
   main.exe cleanup -

USAGE:
   main.exe cleanup [command options] [arguments...]

OPTIONS:
   --server value           [$MINIO_SERVER]
   --access-key value       [$MINIO_ACCESS_KEY]
   --secret-key value       [$MINIO_SECRET_KEY]
   --bucket value          (default: "backups") [$MINIO_BUCKET]
   --db value              The name of the database backup files (my-db-2017-12-19.backup would use 'my-db')
   --keep value, -k value  @7d/1d will keep a backup every 1d for all backups 7d old or older

Configuration

You can configure command line options using environment variables if you wish.

MINIO_SERVER=https://play.minio.io

The Minio server you wish to send backups to.

MINIO_ACCESS_KEY=minio

The Access Key used to connect to your Minio server.

MINIO_SECRET_KEY=miniosecret

The Secret Key used to connect to your Minio server.

MINIO_BUCKET=backups

The Minio bucket you wish to store your backup in.

minback-cleanup's Issues

x509: failed to load system roots and no roots provided

The application running in the container could not validate the certificate for my Minio instance. I added this fix to install the X509 root certs:

From 14f42d2fc17114fc99b99507c5642d7426fa3abb Mon Sep 17 00:00:00 2001
From: Julius Niedworok <[email protected]>
Date: Wed, 31 Mar 2021 19:46:44 +0200
Subject: [PATCH] fix: adding X509 root certificates to validate minio cert

Signed-off-by: Julius Niedworok <[email protected]>
---
 Dockerfile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 176f08c..fd4bba1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -19,7 +19,7 @@ RUN go build -o bin/minback-cleanup -a -installsuffix cgo -ldflags "-s -X main.v
 FROM alpine:latest
 LABEL maintainer="Benjamin Pannell <[email protected]>"
 
-RUN apk add --update tini
+RUN apk add --update tini ca-certificates
 ENTRYPOINT ["/sbin/tini", "--"]
 
 COPY --from=0 /go/src/github.com/SierraSoftworks/minback-cleanup/bin/minback-cleanup /bin/minback-cleanup
@@ -34,4 +34,5 @@ ENV MINIO_ACCESS_KEY=""
 ENV MINIO_SECRET_KEY=""
 
 ENTRYPOINT ["/bin/minback-cleanup"]
-CMD ["cleanup"]
\ No newline at end of file
+CMD ["cleanup"]
+
-- 
2.31.1

If you give me write access to the repository than I'm happy to create a pull request as well.

Cheers,
Julius