oxplot / rsyncbtrfs Goto Github PK

Hi,

Instead of doing this

  trap "
    umount "'"'"$TMP_MOUNT"'"'" &>/dev/null;
    rmdir "'"'"$TMP_MOUNT"'"'";
    btrfs subvolume delete "'"'"$TMP_INPROG/vol"'"'" &>/dev/null;
    rm -f "'"'"$TMP_INPROG/cur"'"'";
    rmdir "'"'"$TMP_INPROG"'"'";
  " INT QUIT TERM EXIT

it can be simplified to this:

  trap '
    umount "$TMP_MOUNT" &>/dev/null
    rmdir "$TMP_MOUNT"
    btrfs subvolume delete "$TMP_INPROG/vol" &>/dev/null
    rm -f "$TMP_INPROG/cur"
    rmdir "$TMP_INPROG"
  ' INT QUIT TERM EXIT

First, the semicolons are not needed.
Second, the explosion of quotes can be reduced to, for example, "$TMP_MOUNT". Notice that the environment variables are known when the trap is executed.

One should be able to somehow manage the number of backups, either by giving a simple number or a maximum age or the minimum of free space to be left.

rsyncbtrfs should then delete the corresponding subvolumes.

If rsync fails, then /cur is inconsistent. Consequently the latest complete update is not in /cur but in the second to last snapshot. This is confusing. The script would be better if the rsyncbtrfs did:

1. create a new snapshot based on cur
2. rsync the data into that new snapshot
3. rename the new snapshot as volume cur

Let the user choose to mount the source location as RO before backing up in order to ensure no files are modified while backing up.

This is one of the requests from #2

rsync allows specifying multiple source paths:

rsync [OPTION...] SRC... [DEST]

This would be useful also for rsyncbtrfs, if we wanted to include multiple different paths into one backup subvolume. Unfortunately, with the way rsyncbtrfs wraps the rsync call, this is not possible. Well, almost... we could use the totally unintuitive call

rsyncbtrfs backup src1 dst src2 [OPTIONS...]

which would call:

rsync [OPTIONS...] src2 src1/ tmp_dst/vol

Note the slash behind src1, which means that the hierarchy is scrambled.

Are there some rsync options I'm not aware of to allow specifying multiple source paths, or would it be necessary to change rsyncbtrfs?

Hello:
This is more of a feature request than an issue, but in order to use this script on a long-term basis it would make sense to be able to specify the intervals over which a user wants to save snapshots. If I am running this script weekly, at some point I am going to want to remove old snapshots, else my disk will fill up.
I would be interested in having this functionality implemented. I have been using a relatively complicated shell script that uses the "cp -a" option for incremental backups. BTRFS is more appealing because of compression and the snapshot formalism. Unfortunately I am not very good with bash, so I cannot implement this myself.

Nasser

hi,
have you never thinked about the idea of add a retention policy for backup?

for example, if you run the script without any argument, it works normally...if you add a parameter ( eg, X ) script will work and if it counts X subvolumes, it will create new subvolume and remove le oldest one.

Hello Mansour:

I hope things are going well. I have been using your Python script for my backups for almost 2 years now and everything is going well.

I have learned more about btrfs, and many new features have been added, which got me thinking:

1. Your script is quite useful and is a simple but more effective solution (IMO) than what the btrfs developers point users to:

https://btrfs.wiki.kernel.org/index.php/UseCases#How_can_I_use_btrfs_for_backups.2Ftime-machine.3F

I think you should contact them and get your script added to the list.

2. One minor issue I tend to run into is that the source filesystem (one being backed up) is typically mounted RW and not RO. I solve this problem by creating a RO snapshot (the source is also btrfs) working from that. It might be nice to add this to your script; the user can specify that the source filesystem is btrfs.

3. I know this might be inconvenience because of the name of the script, but btrfs now supports send/recv functionality that could perform incremental backup without rsync:

https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

it would be nice to make this an option.

I know all of this is alot of work, but I think this is a great script and item #3 would greatly increase the user-base. I am willing to contribute some money to have items 2-3 implemented since I am sure you have other tasks competing for your time, and time is money!

Nasser

Some users might want to backup a tree with every mount point under it.

We should probably modify the default behavior to no-bind-mount. Then the user can pass a argument like --bind-mount.

Hi,

The &> syntax is fine for bash, but not on systems that have /bin/sh -> /bin/dash (e.g. default Ubuntu).

I have two commits for you if you want to. See [email protected]:keestux/rsyncbtrfs.git

BTW. I hate the github pull requests because they don't like "fast forward" merging.

Hi,

from your description:

You can use --bind-mount argument when backing up to instruct rsyncbtrfs to bind mount the source directory under a temp path. This is useful when you don't want to backup all the mount points under the source:

Why not just use the rsync option:

-x, --one-file-system don't cross filesystem boundaries

Basically I think this does the same with less overhead.

See https://btrfs.wiki.kernel.org/index.php/Incremental_Backup

This is one of the requests from #2

So I have yet to get rsyncbtrfs to make a backup that doesn't fail. Usually, It appears that temp/cache files that change on disk or disappear before rsync finishes is the culprit. I've tried 4 backup now and each time it fails with something like:

nmartin@deb:~/Downloads$ rsyncbtrfs backup /home/nmartin /media/backup3tb/nmartin
Create subvolume '/media/backup3tb/nmartin/.inprog-PmnUqyW/vol'
file has vanished: "/home/nmartin/.local/share/gvfs-metadata/label-ARCH_201504-291a92f3.log"
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1070) [sender=3.0.9]
rmdir: failed to remove `/media/backup3tb/nmartin/.inprog-PmnUqyW': Directory not empty

this specific failure was due to a usb pen drive being unplugged during the backup. I guess I could build an excludes file to pass to the backup script, but is there a better method to detect 'OK' errors from rsync vs critical errors?