Backup functionality - managing snapshots automatically about rsyncbtrfs

[oxplot]

If I understand you correctly, you want to be able to delete all the snapshots older than a specific age (e.g. 1 year). Is that right?

[nasserma]

I think for basic functionality, it would make sense to specify an option that maintains a certain number of snapshots. Thus if I am running my rsyncbtrfs backup weekly, I can specify for the script to maintain N backups, which would corresponds to backups for the last N weeks. This does not have to be rigorous, just based on the number of existing backups. Thus if I am running it biweekly or monthly or over any interval, my N backups will be constant.

Ideally, it would be desirable to specify the number of snapshots based upon some interval DWMY, where 0211 would correspond to keeping 0 daily snapshots, 2 snapshots of the past two weeks, 1 snapshot of the past month, and 1 of the past year. I can send you copies of my shell script (modified from one I found on the web) that does this. Although in my script it depends on running the script daily (using cron).

[oxplot]

OK, I came up with a generalisation of the "ideal" solution:
There will be an "organise" command that will keep certain snapshots based on the given rule and discard the rest. A command line example of organise command would be:

rsyncbtrfs organise /backup /backup/home 1D,7 1W,4 1M,12 1Y,10

First two arguments define the location and mount point as usual. The arguments that follow each define an interval and a count in that order. For instance, first one defines an interval of 1 day and a count of 7. This rule translates into: keep the latest backup in the intervals of 1 day for the past 7 day. This simply means, keep backups for past 7 days. The second rule would translate into: keep the latest backups in the intervals of 1 week for the past 4 weeks. It should be clear what the other two rules mean.

Intervals can be defined as multiples of second, minute, hour, day, week, month, quarter and year.

[nasserma]

This would be ideal, given that the user need only specify interval/count pairs corresponding the what they want. Thus if they only want a daily and monthly backup then:

rsyncbtrfs organise /backup /backup/home 1D,7 1W,4

would be sufficient and it would be assumed that this corresponds to:

rsyncbtrfs organise /backup /backup/home 1D,7 1W,4 0M,0 0Y,0

or something along those lines. Please let me know how to send you my own scripts which do something like this, but without command line parsing (you need to edit them explicitly). This should save you some time if you want to implement this, I could try doing it myself, but as I mentioned I have very little experience with bash.

[oxplot]

I've been wanting to polish this script for a while, so I'll have a go at this myself. Thanks for bringing this up.

[oxplot]

This is now implemented in python-based branch with commit bb3edf7 and e152f7c
Closing the issue.

[nasserma]

That's fantastic, I will switch over to the Python implementation and test it out. If there are any other issues or problems, I should be able to address them myself and send a patch.

[nasserma]

I switched to the python implementation and edited my crontab to look like this:

0 0 * * * /root/bin/rsyncbtrfs backup /media/backup /media/backup/home /home
0 1 * * 7 /root/bin/rsyncbtrfs organize /media/backup /media/backup/home 1D7 1W1 1M1 1Y1

so now I should have a backup for the past 7 days, a backup for the week before that, the month before that, and the year before that.

So far so good, although I will check next week to confirm if the deletion is occurring properly.

Thanks!

[oxplot]

nasserma,

I just noticed a few problems with the script and made some changes so that your cron jobs run without problems later. So just grab the latest commit. The command line arguments are the same. You only need to add a "-s" before the commands (e.g. /root/bin/rsyncbtrfs -s organize) to suppress all the messages and confirmations otherwise your cron process might get stuck. Read the help to get more details.