rdiff-backup / rdiff-backup

Reverse differential backup tool, over a network or locally.

website . download . community

rdiff-backup

rdiff-backup is a simple backup tool which can be used locally and remotely, on Linux and Windows, and even cross-platform between both. Users have reported using it successfully on FreeBSD and MacOS X.

Beside its ease of use, one of the main advantages of rdiff-backup is that it does use the same efficient protocol as rsync to transfer and store data. Because rdiff-backup only stores the differences from the previous backup to the next one (a so called reverse incremental backup), the latest backup is always a full backup, making it easiest and fastest to restore the most recent backups, combining the space advantages of incremental backups while keeping the speed advantages of full backups (at least for recent ones).

If the optional dependencies pylibacl and pyxattr are installed, rdiff-backup will support Access Control Lists and Extended Attributes provided the file system(s) also support these features.

1. INSTALLATION

In older Linux distributions the rdiff-backup versions are of the 1.x series, which is not recommended for new installs anymore. Follow the instructions below to install the latest 2.x release of rdiff-backup.

Caution

rdiff-backup 1.x and 2.x aren’t compatible and can’t be mixed in server/client mode! See how to migrate side-by-side.

1.1. Ubuntu Focal or Debian Bullseye or newer (has 2.0)

sudo apt install rdiff-backup

1.2. Ubuntu backports for older versions (needs a backported 2.0)

sudo apt install software-properties-common
sudo add-apt-repository ppa:rdiff-backup/rdiff-backup-backports
sudo apt update
sudo apt install rdiff-backup

1.3. RHEL 7 and its replicas (from EPEL)

sudo yum install epel-release
sudo yum install rdiff-backup
sudo yum install py3libacl pyxattr

Note	the last line is optional to get ACLs and EAs support.

1.4. RHEL 8 and its replicas (from EPEL)

sudo dnf install epel-release
sudo dnf --enablerepo=PowerTools install rdiff-backup

Note

you can add the option --setopt=install_weak_deps=False to the last line if you don’t need ACLs and EAs support. You can install python3-pylibacl and python3-pyxattr also separately. Under RHEL, the repo to enable is codeready-builder-for-rhel-8-x86_64-rpms in order to get access to pyxattr, instead of PowerTools.

Note	This does not enable updates for PowerTools, check the distribution documentation for details on how to do this.

1.5. Fedora 34+

sudo dnf install rdiff-backup

Note	for earlier versions, see the COPR instructions below.

1.6. Debian and derivatives, Raspbian, etc. (from PyPi)

sudo apt install python3-pip python3-setuptools python3-pylibacl python3-pyxattr
sudo pip3 install rdiff-backup

Note	If your platform is not i386 or amd64, e.g. ARM/MIPS/etc, you may need the build dependencies build-essentials, librsync-dev.

1.7. CentOS and RHEL 6 (from PyPi)

sudo yum install centos-release-scl
sudo yum install rh-python36 gcc libacl-devel
scl enable rh-python36 bash
sudo pip install rdiff-backup pyxattr pylibacl  # or rdiff-backup[meta]
echo 'exec scl enable rh-python36 -- rdiff-backup "$@"' | sudo tee /usr/bin/rdiff-backup
sudo chmod +x /usr/bin/rdiff-backup

1.8. Fedora and derivatives (from PyPI)

sudo dnf install python3-pip python3-setuptools py3libacl python3-pyxattr
sudo pip3 install rdiff-backup

1.9. Other Linux and UN*X-oid systems, e.g. BSD (from PyPi)

You need to make sure that the following requirements are met, using your system’s package/software/application manager, searching there for the following keywords:

• Python, 3.9 or higher

• Python pip or pip3, e.g. with python -m ensurepip --upgrade

• librsync or librsync2, 2.0.0 or higher

• libacl or libacl1 or simply acl (optional, to support ACLs)

• SSH, generally OpenSSH, client and/or server (optional, for remote operations)

Then you should only need to call one of the following before you can use rdiff-backup:

sudo pip3 install rdiff-backup        # without optional dependencies
sudo pip3 install rdiff-backup[meta]  # with support for metadata, ACLs and EAs

Note	especially if your platform is not i386 or amd64, e.g. ARM/MIPS/PowerPC/etc, but also if the pip3 installation fails with #include […​].h files missing, you may need the build dependencies named like python3-devel, librsync-dev or libacl1-dev.

1.10. Windows

Just download and unpack the file rdiff-backup-VERSION.win64exe.zip (or win32 if need be). It is available as asset attached to one of the releases available in the releases section. Then drop the binary rdiff-backup.exe somewhere in your PATH and it should work, as it comes with all dependencies included.

Note	starting with rdiff-backup 2.1.1 embedding Python 3.10, rdiff-backup cannot be used on Windows 7 or earlier.

Another way to install rdiff-backup is to use Chocolatey and call choco install rdiff-backup.

For remote operations, you will need to have an SSH package installed. The standard one provided by Microsoft is probably your safest choice, else we recommend using OpenSSH from mls-software.com.

2. BASIC USAGE

Creating your first backup is as easy as calling rdiff-backup <source-dir> <backup-dir> (possibly as root), e.g. rdiff-backup -v5 /home/myuser /run/media/myuser/MYUSBDRIVE/homebackup would save your whole home directory (under Linux) to a USB drive (which you should have formatted with a POSIX file system, e.g. ext4 or xfs). Without the -v5 (v for verbosity), rdiff-backup isn’t very talkative, hence the recommendation.

Subsequent backups can simply be done by calling exactly the same command, again and again. Only the differences will be saved to the backup directory.

If you need to restore the latest version of a file you lost, it can be as simple as copying it back using normal operating system means (cp or copy, or even pointing your file browser at the backup directory). E.g. taking the above example cp -i /run/media/myuser/MYUSBDRIVE/homebackup/mydir/myfile /home/myuser/mydir/myfile and the lost file is back!

There are many more ways to use and tweak rdiff-backup, they’re documented in the man pages, in the documentation directory, or on our website.

3. TROUBLESHOOTING

If you have everything installed properly, and it still doesn’t work, see the enclosed FAQ, the rdiff-backup web page and/or the rdiff-backup-users mailing list.

We’re also happy to help if you create an issue to our GitHub repo. The most important is probably to explain what happened with which version of rdiff-backup, with which command parameters on which operating system version, and attach the output of rdiff-backup run with the very verbose option -v9.

The FAQ in particular is an important reference, especially if you are using smbfs/CIFS, Windows, or have compiled by hand on Mac OS X.

4. CONTRIBUTING

Rdiff-backup is an open source software developed by many people over a long period of time. There is no particular company backing the development of rdiff-backup, so we rely very much on individual contributors who "scratch their itch". All contributions are welcome!

There are many ways to contribute:

• Testing, troubleshooting and writing good bug reports that are easy for other developers to read and act upon

• Reviewing and triaging existing bug reports and issues, helping other developers focus their efforts

• Writing documentation (e.g. the man page), or updating the webpage rdiff-backup.net

• Packaging and shipping rdiff-backup in your own favorite Linux distribution or operating system

• Running tests on your favorite platforms and fixing failing tests

• Writing new tests to get test coverage up

• Fixing bug in existing features or adding new features

If you don’t have anything particular in your mind but want to help out, just browse the list of issues. Both coding and non-coding tasks have been filed as issues.

For source code related documentation see docs/DEVELOP.adoc

4.1. Installing latest development release

To provide meaningful bug reports and help with testing, please use the latest development release.

4.1.1. Ubuntu and Debian development releases

sudo add-apt-repository ppa:rdiff-backup/rdiff-backup-development
sudo apt update
sudo apt install rdiff-backup

4.1.2. Fedora, CentOS and RHEL (from COPR)

On CentOS and RHEL (7 and 8):

sudo yum install dnf-plugins-core epel-release
sudo yum copr enable frankcrawford/rdiff-backup
sudo yum install rdiff-backup

On Fedora 30+:

sudo dnf install dnf-plugins-core
sudo dnf copr enable frankcrawford/rdiff-backup
sudo dnf install rdiff-backup

4.1.3. PyPi pre-releases

sudo pip3 install rdiff-backup --pre

5. Packaging status in distros

rdiff-backup's Issues

[ENH] parallelize the backup process with a job # parameter

Would be nice if there were a -j $(nproc) option.

-j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run simultaneously.

One can wrap many backups in xargs -P $(nproc) but that won't help for single large backups, and 8 is a lot faster than 1.

--max-procs=max-procs
-P max-procs
Run up to max-procs processes at a time; the default is 1.

special characters in directories fail while they work in filenames [WIN->LINUX]

I am running rdiff-backup 1.3.3 w32 (binaries from the official homepage) to backup remotely to a linux box (1.3.3 version from the homepage as well). I tested directories and files where some included (german) special characters and found that rdiff-backup fails on directories including special characters (while files including their content are correctly processed). local tests from windows to windows and from linux to linux both worked for all special characters.

As you can see in the last logline before the exception in the message the path is correctly converted while in the actual path it lacks a conversion from the byte-string to an unicode string. This difference is probably because in the Log class (log.py) there is some code for doing this conversion. Maybe someone with more insight into the rdiff-backup code could help me figure out a solution for this?

The logfile (-v9) snippet where the error occurs:

Thu Mar 15 21:47:35 2018 Processing changed file .
Thu Mar 15 21:47:35 2018 Processing changed file AAalpha
Thu Mar 15 21:47:35 2018 Making directory path /media/backup/ilen/test-st/rdiff-backup-data/increments
Thu Mar 15 21:47:35 2018 Making directory /media/backup/ilen/test-st/AAalpha
Thu Mar 15 21:47:35 2018 Processing changed file AAalpha/ABNVääääüü.txt
Thu Mar 15 21:47:35 2018 Making directory path /media/backup/ilen/test-st/rdiff-backup-data/increments/AAalpha
Thu Mar 15 21:47:35 2018 Regular copying ('AAalpha', 'ABNV\xe4\xe4\xe4\xe4\xfc\xfc.txt') to /media/backup/ilen/test-st/AAalpha/rdiff-backup.tmp.1
Thu Mar 15 21:47:35 2018 Writing file object to /media/backup/ilen/test-st/AAalpha/rdiff-backup.tmp.1
Thu Mar 15 21:47:35 2018 Copying attributes from ('AAalpha', 'ABNV\xe4\xe4\xe4\xe4\xfc\xfc.txt') to /media/backup/ilen/test-st/AAalpha/rdiff-backup.tmp.1
Thu Mar 15 21:47:35 2018 Setting time of /media/backup/ilen/test-st/AAalpha/rdiff-backup.tmp.1 to 1521147112
Thu Mar 15 21:47:35 2018 Renaming /media/backup/ilen/test-st/AAalpha/rdiff-backup.tmp.1 to /media/backup/ilen/test-st/AAalpha/ABNVääääüü.txt
Thu Mar 15 21:47:35 2018 Processing changed file ABCÖ
Thu Mar 15 21:47:35 2018 Copying attributes from ('AAalpha',) to /media/backup/ilen/test-st/AAalpha
Thu Mar 15 21:47:35 2018 Setting time of /media/backup/ilen/test-st/AAalpha to 1521147107
Thu Mar 15 21:47:35 2018 Making directory /media/backup/ilen/test-st/ABCÖ
Thu Mar 15 21:47:35 2018 Sending back exception [Errno 2] No such file or directory: '/media/backup/ilen/test-st/ABC\xd6' of type <type 'exceptions.OSError'>:
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/connection.py", line 335, in answer_request
result = apply(eval(request.function_string), argument_list)
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 240, in patch
ITR(diff.index, diff)
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in call
branch.start_process(*args)
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 630, in start_process
if diff_rorp.isdir(): self.prepare_dir(diff_rorp, self.base_rp)
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 658, in prepare_dir
base_rp.mkdir()
File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
self.conn.os.mkdir(self.path)

Create a PPA for Ubuntu testing

We should have a PPA on Launchpad for rdiff-backup so it would be easy for people to test the latest version.

I am assigning this myself, the issue is just a reminder.

I have contacted the Launchpad user @hstock to grant me access to https://launchpad.net/rdiff-backup so I can publish there an official PPA (instead of just using https://launchpad.net/~otto/+archive/ubuntu/ppa/+builds?build_text=&build_state=all)

Create a CI pipeline

We should create a CI pipeline that runs on every commit and every pull request.

What tests to run?

The new Python 3 branch introduces the Tox testing framework, so we should at least run Tox.

Tox should be extended to test as much as possible. We need to look into if Tox is enough, or if we need some separate tool to do acceptance tests and run in bash some actual rdiff-backup jobs/cycles. I would be interested in having some tests that generate a backup archive with version 1.2.8 and then runs the latest git development version to further process that archive and make new increments etc. This would show us if the code remains backwards compatible or not.

We could also run some static code analysis. Since this is a Python project, running flake8 on all Python files would seem natural. Requiring everybody to follow PEP-8 in their Python code is probably OK.

What testing system to use?

Github does not have any built in testing system (unlike Gitlab which has Gitlab CI). Personally I have lot's of experience with Travis-CI and it is free for open source projects, so we could adopt it.

vfat misidentified as case sensitive

I have just noticed some trouble with rdiff-backup on vfat (as target fs) in linux. The filesystem is not completely case-insensitive when mounted with the option iocharset=utf8.
$ touch A ; touch a
will result in an error, but
$ touch a; touch A
will not.

$ stat a
will not recognize the file A, but
$ touch a
is not allowed to create the file a if A exists.

rdiff-backup should always assume case insensitive behavior on vfat! If not, you will run into trouble, as soon as there are two files with the same lowercase representation. Currently, rdiff-backup mistakes vfat as case sensitive!

Originally posted by @LeonardMichlmayr in #11 (comment)

Broken links to www.nongnu.org

In the README there are many links that point to the www.nongnu.org domain.
This domain appears to return 404 only:

~> curl -I http://www.nongnu.org/rdiff-backup/
HTTP/1.1 404 Not Found
(...)

But it works if we omit the www.
Is this a temporary issue with www.nongnu.org? I can submit a PR if URL should be changed to not use www.

rdiff-backup lacks sparse file support

Today I noticed something was off with the sizes of some of our backups: a VM with a 20GB hard drive had a 450+GB backup. Turns out /var/log/lastlog is a 458GB sparse file on a number of systems and rdiff-backup backs up the whole thing.

A quick Google suggests this was discovered in 2005 already, and a patch was written in 2011, but unfortunately due to maintainer sleep this never got into rdiff-backup code. May be useful to integrate in the repository now?

If it helps I could make a pull request but as it's posted there it's not actually my code...

githubpage / README looks really technical

Thanks for taking over the project. I do not know if you can tell yourself, thats why I fill in this issue. If the github repo is going to be the new frontpage for the project, than it must be improved radically. It is very technical now - could you instead replace it by something more like http://www.nongnu.org/rdiff-backup/index.html with links to faq, installation etc? Btw installation can be as easy as aptitude install rdiff-backup.

Option remove-older-than should have ability to remove only sub-sets of repo history

Hi,

is in this GitHub repo included patch from "Josh Nisly" from this issue: http://savannah.nongnu.org/bugs/?25962 ?

And is there any plan to release next official version? Latest is 1.3.3. I'm a Gentoo user :)

Thank you very much. Is great, that you take next rdiff-backup development.

Split gh-pages branch to separate repository

Currently our webpage is built from the gh-pages branch of this same repository. I don't think that makes sense in the long run. We should consider splitting that out into a separate repository at github.com/rdiff-backup/website or something along those lines.

Fail when renaming with different case on case insensitive file system

I'm using rdiff-backup 1.2.8 to backup files on a Windows machine with cygwin. The source is the local hard drive and the backup destination is an external hard drive, both NTFS. Renaming a folder called archive to Archive causes an AssertError assert not incrp.lstat(), incrp. This has been reported before: https://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/case-insensitive-filesystems-assert-not-incrp-lstat-100362/

Renaming the folder to its original name works after running rdiff-backup --check-destination-dir "$dst".

I wrote a script to reproduce that error:

#!/bin/sh

src="$(mktemp -d)"
dst="$(mktemp -d)"
echo "backup source: $src"
echo "backup destination: $dst"

mkdir "$src/archive"
rdiff-backup "$src" "$dst"

sleep 1
mv "$src/archive" "$src/Archive-tmp"
mv "$src/Archive-tmp" "$src/Archive"
rdiff-backup "$src" "$dst"

ls -la "$src"
ls -la "$dst"
rm "$src" -rf
rm "$dst" -rf

I ran that script on my linux machine with a FAT32 usb drive: TMPDIR=/media/C808-E853/ sh rdiff-backup-23.sh

It gave me a different error, but the bug is probably related:

backup source: /media/C808-E853/tmp.a0Op9QfBAo
backup destination: /media/C808-E853/tmp.l1QgeTHVzJ
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.l1QgeTHVzJ/rdiff-backup-data
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.l1QgeTHVzJ/rdiff-backup-data
Exception '[Errno 17] File exists: '/media/C808-E853/tmp.l1QgeTHVzJ/Archive'' raised of class '<type 'exceptions.OSError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)
OSError: [Errno 17] File exists: '/media/C808-E853/tmp.l1QgeTHVzJ/Archive'

Exception '' raised of class '<type 'exceptions.AssertionError'>'

I am trying rdiff-backup for the first time. It seems buggy:

~$ uname -a
Linux laptop 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
~ $ rdiff-backup --version
rdiff-backup 1.2.8
~ $ rdiff-backup Documents/Inbox /media/er/Vol2/backup/laptop
Exception '' raised of class '<type 'exceptions.AssertionError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 334, in Backup
    rpout.conn.fs_abilities.backup_set_globals(rpin, force)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 922, in backup_set_globals
    dest_fsa = FSAbilities('destination').init_readwrite(Globals.rbdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 156, in init_readwrite
    self.set_case_sensitive_readwrite(subdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 304, in set_case_sensitive_readwrite
    assert not upper_a.lstat()

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 334, in Backup
    rpout.conn.fs_abilities.backup_set_globals(rpin, force)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 922, in backup_set_globals
    dest_fsa = FSAbilities('destination').init_readwrite(Globals.rbdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 156, in init_readwrite
    self.set_case_sensitive_readwrite(subdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 304, in set_case_sensitive_readwrite
    assert not upper_a.lstat()
AssertionError

/media/er/Vol2/backup/laptop $ ls -R
.:
rdiff-backup-data

./rdiff-backup-data:
rdiff-backup.tmp.0

./rdiff-backup-data/rdiff-backup.tmp.0:

Exception ''ascii' codec can't decode byte 0xe2

Weird charset prob.
I was rdiff-backuping for month via NFS, I recently migrated the rdiff_repo , and now execute rdiff-backup locally (on a synology)
It's right that there are some weird encoded characters in a couple of file names, but rdiff-backup never complained when used via NFS
Is there any way either to convert the files src & dst to prevent this error ? Or Would it be possible to forxce rdiff-backup not to stop on file errors ?

Exception ''ascii' codec can't decode byte 0xe2 in position 9: ordinal not in range(128)' raised of class '<type 'exceptions.UnicodeDecodeError'>':
File "/opt/lib/python2.7/site-packages/rdiff_backup/Main.py", line 306, in error_check_Main
File "/opt/lib/python2.7/site-packages/rdiff_backup/Main.py", line 326, in Main
File "/opt/lib/python2.7/site-packages/rdiff_backup/Main.py", line 282, in take_action
File "/opt/lib/python2.7/site-packages/rdiff_backup/Main.py", line 345, in Backup
File "/opt/lib/python2.7/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
File "/opt/lib/python2.7/site-packages/rdiff_backup/backup.py", line 249, in patch_and_increment
File "/opt/lib/python2.7/site-packages/rdiff_backup/rorpiter.py", line 177, in FillInIter
File "/opt/lib/python2.7/site-packages/rdiff_backup/backup.py", line 103, in get_diffs
File "/opt/lib/python2.7/site-packages/rdiff_backup/backup.py", line 174, in get_sigs
File "/opt/lib/python2.7/site-packages/rdiff_backup/backup.py", line 328, in next
File "/opt/lib/python2.7/site-packages/rdiff_backup/rorpiter.py", line 92, in Collate2Iters
File "/opt/lib/python2.7/site-packages/rdiff_backup/rorpiter.py", line 342, in next
File "/opt/lib/python2.7/site-packages/rdiff_backup/selection.py", line 132, in Iterate_fast
File "/opt/lib/python2.7/site-packages/rdiff_backup/selection.py", line 118, in diryield
File "/opt/lib/python2.7/site-packages/rdiff_backup/selection.py", line 181, in listdir

Fix compilation for librsync >= 1.0.0

I'm the maintainer of rdiff-backup and librsync in Gentoo. I had to use the Fedora patch[1] to fix compilation for librsync >= 1.0.0. Can you please apply this to the next release?

[1] http://pkgs.fedoraproject.org/cgit/rpms/rdiff-backup.git/plain/rdiff-backup-1.2.8-librsync-1.0.0.patch

Current release status

This is really a question and not a bug report.

I see there is a release marked stable (1.2.8) and one unstable (1.3.3); both 7 years old. Should one still go for 1.2.8 vs 1.3.3 vs building from source on master ?

MemoryError exception when remote server does not start

(Note: This bug has been verified on CentOS 7. I have not verified whether it is still present in the github code, but from a quick check the code appears to be the same.)

If rdiff-backup is not installed on remote, ssh returns:

bash: rdiff-backup: command not found

connection.py LowLevelPipeConnection._get reads that text, interprets the text as a binary long, and then calls _read which tries to allocate a huge amount of memory.

            header_string = self.inpipe.read(9)                             
            if not len(header_string) == 9:                                 
                    raise ConnectionReadError("Truncated header string (problem "
                                                                      "probably originated remotely)")
            format_string, req_num, length = (header_string[0],             
                                                                              ord(header_string[1]),
                                                                              C.str2long(header_string[2:]))

Exception:

Exception '' raised of class '<type 'exceptions.MemoryError'>':             
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)                                                      
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)                          
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)                                     
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)                              
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')                       
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 453, in __call__
    return apply(self.connection.reval, (self.name,) + args)                
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 371, in reval
    result = self.get_response(req_num)                                     
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 318, in get_response
    try: req_num, object = self._get()                                      
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 243, in _get
    data = self._read(length)                                               
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 211, in _read
    try: return self.inpipe.read(length)    

Traceback (most recent call last):                                          
  File "/usr/bin/rdiff-backup", line 30, in <module>                        
    rdiff_backup.Main.error_check_Main(sys.argv[1:])                        
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)                                                      
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 321, in Main
    rps = map(SetConnections.cmdpair2rp, cmdpairs)                          
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 76, in cmdpair2rp
    if cmd: conn = init_connection(cmd)                                     
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 150, in init_connection
    check_connection_version(conn, remote_cmd)                              
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/SetConnections.py", line 158, in check_connection_version
    try: remote_version = conn.Globals.get('version')                       
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 453, in __call__
    return apply(self.connection.reval, (self.name,) + args)                
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 371, in reval
    result = self.get_response(req_num)                                     
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 318, in get_response
    try: req_num, object = self._get()                                      
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 243, in _get
    data = self._read(length)                                               
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/connection.py", line 211, in _read
    try: return self.inpipe.read(length)                                    
MemoryError 

It looks like this is an old bug, previous reports include:

• mailing list
• on launchpad
• savannah
• serverfault
• mailing list
• redhat bugzilla

[Feature request] Split a backup directory

This is something that in my opinion comes up pretty often. For example, in case your directory structure changes. As far as I know rdiff-backup won't recognize that I moved something. However, it should be relatively easy to split up the backup directory (metadata, diffs and actual files) into 2 backup directories. Use case: Move huge files to another partition (or filesystem).

This would be useful to avoid copying huge amounts of data or removing single directories or files with all their history from the larger backup directory.

Merging repositories is probably not as easy as that (since backup times probably do not overlap). But manual renaming or moving of a directory should be fairly easy (although it probably will change some of the attributes). Use case: Change the name or location of a directory or file manually. E.g., moving a folder with a lot of content to another location within the source directory. Then you could perform this change manually.

Remove os_walk function from rdiff_backup/FilenameMapping.py

Issue by ericzolf
Sunday Apr 14, 2019 at 09:04 GMT
Originally opened as ericzolf#2

---

The function was only introduced to support Python 2.2 where os.walk didn't yet exist, and can be removed.

UnicodeDecodeError with WinACLs (locally executed; Windows)

Executing the following command results in an exception (see below):

rdiff-backup -v4 --print-statistics F:/test-backup F:/test-backup-remote

[...]
Starting mirror F:/test-backup to F:/test-backup-remote
Exception ''ascii' codec can't decode byte 0xc3 in position 12: ordinal not in range(128)' raised of class '<type 'exceptions.UnicodeDecodeError'>':
File "F:...\lib.win32-2.7\rdiff_backup\Main.py", line 306, in error_check_Main
try: Main(arglist)
File "F:...\lib.win32-2.7\rdiff_backup\Main.py", line 326, in Main
take_action(rps)
File "F:...\lib.win32-2.7\rdiff_backup\Main.py", line 282, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "F:...\lib.win32-2.7\rdiff_backup\Main.py", line 348, in Backup
backup.Mirror(rpin, rpout)
File "F:...\lib.win32-2.7\rdiff_backup\backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "F:...\lib.win32-2.7\rdiff_backup\backup.py", line 240, in patch
ITR(diff.index, diff)
File "F:...\lib.win32-2.7\rdiff_backup\rorpiter.py", line 281, in call
last_branch.fast_process(*args)
File "F:...\lib.win32-2.7\rdiff_backup\backup.py", line 533, in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
File "F:...\lib.win32-2.7\rdiff_backup\backup.py", line 563, in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
File "F:...\lib.win32-2.7\rdiff_backup\rpath.py", line 192, in copy_attribs
if Globals.win_acls_write: rpout.write_win_acl(rpin.get_win_acl())
File "F:...\lib.win32-2.7\rdiff_backup\rpath.py", line 1417, in write_win_acl
write_win_acl(self, acl)
File "F:...\lib.win32-2.7\rdiff_backup\win_acls.py", line 254, in rpath_set_win_acl
acl.from_string(acl_str)
File "F:...\lib.win32-2.7\rdiff_backup\win_acls.py", line 207, in from_string
else: self.index = tuple(unicode(C.acl_unquote(filename)).split('/'))

adding the --no-acls option works around this issue

Crash with AssertionError while backing up to exFAT partition

Using rdiff-backup 1.2.8 (Ubuntu / Debian repositories) and 1.3.3 (built from source), when backing up to an exFAT formatted partition it crashes. From the crash output, it is related to this line:

rdiff_backup/fs_abilities.py:
assert not upper_a.lstat()

https://github.com/sol1/rdiff-backup/blob/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/rdiff_backup/fs_abilities.py#L325

If I comment out that line (so no assert command is run), then rdiff-backup will complete without error using an exFAT partition. I do not understand why rdiff-backup crashes on exFAT but other non-case-sensitive fomats such as FAT32 and NTFS do not crash. I wonder if that line is essential or what the implication would be if it were removed?

The full error is as follows (from source at savannah it is line 304, instead of line 325 here, since I was not successful in building from source with the current github code):

Message: Found interrupted initial backup. Removing...
Exception '' raised of class '<type 'exceptions.AssertionError'>':
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 306, in error_check_Main
    try: Main(arglist)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 326, in Main
    take_action(rps)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 282, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 336, in Backup
    rpout.conn.fs_abilities.backup_set_globals(rpin, force)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 959, in backup_set_globals
    dest_fsa = FSAbilities('destination').init_readwrite(Globals.rbdir)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 156, in init_readwrite
    self.set_case_sensitive_readwrite(subdir)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 304, in set_case_sensitive_readwrite
    assert not upper_a.lstat()

Traceback (most recent call last):
  File "/usr/local/bin/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 306, in error_check_Main
    try: Main(arglist)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 326, in Main
    take_action(rps)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 282, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 336, in Backup
    rpout.conn.fs_abilities.backup_set_globals(rpin, force)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 959, in backup_set_globals
    dest_fsa = FSAbilities('destination').init_readwrite(Globals.rbdir)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 156, in init_readwrite
    self.set_case_sensitive_readwrite(subdir)
  File "/usr/local/lib/python2.7/dist-packages/rdiff_backup/fs_abilities.py", line 304, in set_case_sensitive_readwrite
    assert not upper_a.lstat()
AssertionError

Decide where to store test files

The next Tox framework uses test files from https://github.com/ericzolf/rdiff-backup/releases

Where should we host this in the future?

Options:

• in gh-pages branch as a tar.gz for download via rdiff-backup.net/testfiles.tar.gz to keep the master branch / main repo "clean" from binary stuff
• in master branch as a tar.gz and ensuring the test suite and the test data evolve lockstep in the same commits
• somewhere else?

Alternatively, do we even have to have the files as a binary package? Could it be stored in an uncompressed plain files way in the master branch and then some kind of test-prepare.sh script would set the file permissions and attributes (since git does not store them and the special attributes is required for the test files, as that is almost the whole point in them).

ListError prints wrong path when excluding user mounted directory as root

Running as root, trying to exclude a user-mounted directory, rdiff-backup prints a ListError with a confusing path. Here's a script that shows the issue. Comments in the bash script explain what I've found.

listerror.sh.txt

Decide and implement a generic documentation handling method

The problem is bigger than initially described (see below the line): we need to streamline the distribution process, also when it comes to documentation: we have currently a mixture of txt, md and html, and we should all align and handle similarly.

---

During installation:

sudo python3 dist/setup.py install --prefix=/usr/local
[...]
running install_data
error: can't copy 'FAQ.html': doesn't exist or not a regular file

FAQ-body.html does exist but not FAQ.html, which is actually created in dist/makedist with the function MakeHTML("FAQ", "rdiff-backup FAQ").

Change permission/mode of created files

I want to backup my files to a remote server.
On the server the permissions are supposed to be 0750 (rwxr-x---) for directories and 0740 (rwxr-----) for files.
But when I restore my files I want to retain their original permissions which is done by mirror_metadata.*.snapshot how I understand it.
So I would like if their is any option besides running rdiff-backup and then find -type d -exec chmod 0750 {} \; && find -type f -exec chmod 0740 {} \;.

And if such an option does not exist I would like to propose:

• --mirror-file-mode perm: set the permissions of the mirror files to the given ones
• --mirror-directory-mode perm: set the permissions of the mirror directories to the given ones
• --mirror-mode perm: This is short for --mirror-file-mode perm --mirror-directory-mode perm
• --mirror-owner uid_or_uname:gid_or_gname: set the permissions of the mirror file to the given ones

Clean-up the code with pep8 and replace tabs with spaces

The code looks strange and more difficult to read and edit currently. That would be my next task after having got the bytes MR merged into master.

Cannot build from a git clone

The repository layout is different from the tarball, so setup.py is completely broken when run from a git clone of this repository.

> python setup.py --librsync-dir="$lab/tools/librsync/build" install --prefix=~/.local
running install
running build
running build_py
error: package directory 'rdiff_backup' does not exist

I tried soft links to help the script find things, but it didn't work.

IOError: [Errno 20] Not a directory

I’m doing separate backups of several directories to a single medium. The cron task executing the backups recently started sending me the following output for two of those directories:

Exception '[Errno 20] Not a directory' raised of class '<type 'exceptions.IOError'>':
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 306, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 326, in Main
    take_action(rps)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 282, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 345, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 251, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 727, in start_process
    inc.fsync_with_dir() # must write inc before rp changed
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1315, in fsync_with_dir
    if Globals.fsync_directories: self.get_parent_rp().fsync()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1138, in get_parent_rp
    return self.__class__(self.conn, self.base, self.index[:-1])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 884, in __init__
    else: self.setdata()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 909, in setdata
    if self.lstat(): self.conn.rpath.setdata_local(self)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1502, in setdata_local
    if Globals.acls_conn: rpath.data['acl'] = acl_get(rpath)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 580, in rpath_acl_get
    if not rp.issym(): acl.read_from_rp(rp)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 366, in read_from_rp
    rp.conn.eas_acls.get_acl_lists_from_rp(rp)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 399, in get_acl_lists_from_rp
    try: acl = posix1e.ACL(file=rp.path)

Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 306, in error_check_Main
    try: Main(arglist)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 326, in Main
    take_action(rps)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 282, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/Main.py", line 345, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 251, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/backup.py", line 727, in start_process
    inc.fsync_with_dir() # must write inc before rp changed
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1315, in fsync_with_dir
    if Globals.fsync_directories: self.get_parent_rp().fsync()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1138, in get_parent_rp
    return self.__class__(self.conn, self.base, self.index[:-1])
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 884, in __init__
    else: self.setdata()
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 909, in setdata
    if self.lstat(): self.conn.rpath.setdata_local(self)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/rpath.py", line 1502, in setdata_local
    if Globals.acls_conn: rpath.data['acl'] = acl_get(rpath)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 580, in rpath_acl_get
    if not rp.issym(): acl.read_from_rp(rp)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 366, in read_from_rp
    rp.conn.eas_acls.get_acl_lists_from_rp(rp)
  File "/usr/lib64/python2.7/site-packages/rdiff_backup/eas_acls.py", line 399, in get_acl_lists_from_rp
    try: acl = posix1e.ACL(file=rp.path)
IOError: [Errno 20] Not a directory

Or simply such an output without any backtrace:

ListError docs/ruby/classes/REXML/Encoding.src/M006539.html/M006540.html [Errno 20] Not a directory

The output is always different. If it is the latter one, different files are mentioned each time (I didn’t get this error for the same file twice).

I ran --check-destination-dir, but nothing wrong was found in one case, in the other rdiff-backup tried to regress, nonetheless ended up with the error Not a directory again (note: trying to run rdiff-backup-regress.sh is therefore not of much help – I tried that, because --verify didn’t find checksums for many files).

Validate if workaround with import at the end of metadata.py can be removed

Issue by ericzolf
Sunday Apr 14, 2019 at 09:21 GMT
Originally opened as ericzolf#3

---

From the code at the end of rdiff_backup/metadata.py:

from . import eas_acls, win_acls # put at bottom to avoid python circularity bug

Some destination dirs tend to bloat

I have noticed that rdiff-backup is extremely bloaty with one of my destination dirs. To illustrate I ran --increments-size on two different ones:

the bloaty destination directory

Size: 765MB
Cumulative size: 877MB
Number of increments: 2139
du -sh: 11G

a different/normal destination directory

Size: 3.51GB
Cumulative size: 3.99GB
Number of increments: 6986
du -sh: 11G

So it makes something like 10 gigabytes for the rdiff-backup data which seems to me like a lot (well, the other one isn’t lean either, but at least it’s more proportional to the size of actual data). I suspect that there are files that don’t need to be there, but I failed to find out how to distinguish them. Also, --check-destination-dir didn’t do anything.

Cannot build against current librsync version, 2.0.2

Building rdiff-backup 1.2.8:

    _librsyncmodule.c: In function '_librsync_new_sigmaker':
    _librsyncmodule.c:63:17: error: 'RS_DEFAULT_STRONG_LEN' undeclared (first use in this function)
             (size_t)RS_DEFAULT_STRONG_LEN);
                     ^~~~~~~~~~~~~~~~~~~~~
    _librsyncmodule.c:63:17: note: each undeclared identifier is reported only once for each function it appears in
    _librsyncmodule.c:62:17: error: too few arguments to function 'rs_sig_begin'
       sm->sig_job = rs_sig_begin((size_t)blocklen,
                     ^~~~~~~~~~~~
    In file included from _librsyncmodule.c:25:0:
    C:/msys64/mingw64/include/librsync.h:397:11: note: declared here
     rs_job_t *rs_sig_begin(size_t new_block_len, size_t strong_sum_len,
               ^~~~~~~~~~~~

This page suggests the following patch (I haven't tested it myself):

--- _librsyncmodule.c.ORIG 2006-11-11 23:32:01.000000000 -0800
+++ _librsyncmodule.c 2018-02-20 11:22:06.529111816 -0800
@@ -59,8 +59,8 @@
 if (sm == NULL) return NULL;
 sm->x_attr = NULL;

- sm->sig_job = rs_sig_begin((size_t)blocklen,
- (size_t)RS_DEFAULT_STRONG_LEN);
+ sm->sig_job = rs_sig_begin((size_t)blocklen, 8,
+ (size_t)RS_MD4_SIG_MAGIC);
 return (PyObject*)sm;
 }

Set up rdiff-backup on PyPi

The package rdiff-backup is listed at https://pypi.org/project/rdiff-backup/ but no versions after 0.1.1 have been released and the author anyway is unknown. We should try to get access to this package on PyPi and integrate it to be automatically published from tags released on Github.

Use Blake2 instead of MD4 with a transition possibility

As written by Joe:

Per the above patch, rdiff-backup continues to use the old MD4 hashes with librsync1. But my understanding is that the MD4 hash is only used to compare files over the network between 2 instances of rdiff-backup (client and server). The hashes are not incorporated into the .diff files (delta files in librsync parlance) that are stored in the rdiff-backup repository. Therefore, even if rdiff-backup was modified to use the newer Blake2 hashes, rdiff-backup would remain compatible with older backup repositories. Thus, I think it is incorrect to say that 'All backups must therefore start from "scratch"'.

I think it probably would be good if rdiff-backup started using Blake2 rather than MD4 hashes by default. But when communicating with older instances of rdiff-backup which don't use Blake2, rdiff-backup should probably revert to using MD4 rather than failing.

Required changes would involve src/_librsyncmodule.c where RS_MD4_SIG_MAGIC is being used (RS_BLAKE2_SIG_MAGIC could be used instead resp. additionally). I'm wondering if rdiff-backup has already a mean to handle any parameter of the communication, which would work with an older version, but we'll see once we're so far.

Bugreport: "Warning: modification time of XYZ/rdiff-backup.tmp.231316 isbefore 1970

Warning: modification time of /mnt/drive/backup-lasse/Archiv/lasse_home/andreas/Documents/CodeProjects/CCC2011/r0ket/Archiv/original-firmware/filesystem/rdiff-backup.tmp.231316 isbefore 1970
Exception '[Errno 5] Input/output error: '/mnt/drive/backup-lasse/Archiv/lasse_home/andreas/Documents/CodeProjects/Cpp/Experiments/branches/.svn/prop-base'' raised of class '<type 'exceptions.OSError'>':
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in call
branch.start_process(*args)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 626, in start_process
if diff_rorp.isdir(): self.prepare_dir(diff_rorp, self.base_rp)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
base_rp.mkdir()
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
self.conn.os.mkdir(self.path)

Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 30, in
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in call
branch.start_process(*args)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 626, in start_process
if diff_rorp.isdir(): self.prepare_dir(diff_rorp, self.base_rp)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
base_rp.mkdir()
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
self.conn.os.mkdir(self.path)
OSError: [Errno 5] Input/output error: '/mnt/drive/backup-lasse/Archiv/lasse_home/andreas/Documents/CodeProjects/Cpp/Experiments/branches/.svn/prop-base'

CIFS filesystem with symbolic links

I am mounting a windows drive using cifs and try to use rdiff-backup from a linux machine to the cifs mount.

If there are no symbolic links this works fine, but with links I get:

OSError: [Errno 95] Operation not supported

from rpath.py:1002 ( self.conn.os.symlink(linktext, self.path) ).

Also works fine if I pass --exclude-symbolic-links ofc.

After some googling I found an old bug #20342: fails backing up to cifs share on savannah that seemed to fix the same or a similar problem. That fix seem to be cd2cb0a in git.

But that was back in 2007 - any idea of the current status regarding this, should this work now ? ( I am using rdiff-backup 1.2.8 )

Support backup of files changing during backup time

Hi!
For first, thank you for this tool. Saved my ass many times.

I'm facing "Bad index order:" assert exception since last interrupted backup. Since that moment, i cannot run backup correctly and end on this assertion. When I looked around, it seems like there is no clear statement how to fix this except delete backup and rerun it from scratch.

I'm running this on Synology DMS linux with Python 2.7.11 (default, Mar 11 2016, 02:18:13)
admin@Backup03:~$ rdiff-backup-2.6 --version rdiff-backup 1.2.8

Can you help me?

`Previous backup seems to have failed, regressing destination now.
Exception 'Bad index order: ('long_filename_data', '16') >= ('1_firma', '2010', '04_org\xc3\xa1ny spole\xc4\x8dnosti', 'Dozor\xc4\x8dn\xc3\xad rada', 'Volba \xc4\x8dlenky DR', 'RE Datumy do z\xc3\xa1pisu p\xc5\x99estavenstva.msg')' raised of class '<type 'exceptions.AssertionError'>':
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 337, in Backup
backup_final_init(rpout)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 501, in backup_final_init
checkdest_if_necessary(rpout)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
dest_rp.conn.regress.Regress(dest_rp)
File "/opt/lib/python2.5/site-packages/rdiff_backup/regress.py", line 71, in Regress
for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
File "/opt/lib/python2.5/site-packages/rdiff_backup/rorpiter.py", line 275, in call
assert 0, "Bad index order: %s >= %s" % (self.index, index)

Traceback (most recent call last):
File "/bin/rdiff-backup", line 30, in
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 337, in Backup
backup_final_init(rpout)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 501, in backup_final_init
checkdest_if_necessary(rpout)
File "/opt/lib/python2.5/site-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
dest_rp.conn.regress.Regress(dest_rp)
File "/opt/lib/python2.5/site-packages/rdiff_backup/regress.py", line 71, in Regress
for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
File "/opt/lib/python2.5/site-packages/rdiff_backup/rorpiter.py", line 275, in call
assert 0, "Bad index order: %s >= %s" % (self.index, index)
AssertionError: Bad index order: ('long_filename_data', '16') >= ('1_firma', '2010', '04_org\xc3\xa1ny spole\xc4\x8dnosti', 'Dozor\xc4\x8dn\xc3 \xad rada', 'Volba \xc4\x8dlenky DR', 'RE Datumy do z\xc3\xa1pisu p\xc5\x99estavenstva.msg')
`

No error raised in certain occasions when excluding files the wrong way

This bug in the input options check was originally reported on the mailing list here:
https://lists.nongnu.org/archive/html/rdiff-backup-users/2017-01/msg00005.html

In this example the software correctly throws an error, because the excluded file is not specified correctly:

$ mkdir temp
$ touch temp/file.txt
$ touch temp/picture.jpg
$ rdiff-backup --exclude file.txt temp backup
Fatal Error: Fatal Error: The file specification
    'file.txt'
cannot match any files in the base directory
    'temp'
Useful file specifications begin with the base directory or some
pattern (such as '**') which matches the base directory.

But in certain cases this check fails and no error is printed even if the exclusion is not valid, like in this:

$ mkdir temp
$ cd temp
$ mkdir .one
$ mkdir .two
$ mkdir three
$ mkdir four
$ rdiff-backup --exclude .one --exclude ./.two --exclude ./three . ../backup
$ ls -a ../backup
.  ..  four  .one  rdiff-backup-data

or in this:

$ mkdir temp
$ touch temp/file.txt
$ touch temp/temp.txt
$ rdiff-backup --exclude temp.txt temp backup
$ ls backup
file.txt  rdiff-backup-data  temp.txt

Python 3 port

I understand that it's still left unmaintained but I'll just create this issue for anyone to track progress if you don't mind.

#40

Crashing on Exception '[Errno 61] No data available'

Under certain circumstances, rdiff-backup tries to remove an ACLs from the file, while there are already no ACLs.

There is no need to bail on this simple non-fatal condition.

Attached patch made against latest version from Git fixes this. Let me know if you want me to check this into repository.

Thanks,
Ondrej

rdiff-backup-enodata-fix.patch.txt

Create .github/FUNDING.yml file for this repo?

Github supports a built-in "help fund this open source project" button that can be activated via .github/FUNDING.yml. This is not a high priority, but some time later we might want to consider if we want this and also if we organize this project under the umbrella of some existing open source organisation, like SPI or Samba (Samba hosts rsync).

I will write more on this topic later, now I just opened this as a reminder to think about this later.

patch for compare.py - exit code

I have a patch (from compare.py in rdiff-backup 1.2.8) below which I use (and need) so that if a verification run fails with one or more verification hash failures (as opposed to some other sort of glitch/crash) we get a specific exit number (2), and a specific message. The original code gives an exit number equal to the total of verification failures and no specific text - this makes it impossible to be sure of the cause of failure - not least because if the run is aborted (killed) the returned code is 1. Please can this patch be incorporated in future releases:

--- compare.py  2015-11-30 17:12:56.046174495 +0000
+++ compare.py  2015-11-30 17:16:23.978174495 +0000
@@ -101,2 +101,5 @@
-       if not bad_files: log.Log("Every file verified successfully.", 3)
-       return bad_files
+       if bad_files:
+               log.Log("Not all files could be verified.", 3)
+               return 2
+       log.Log("Every file verified successfully.", 3)
+       return 0

Create a CONTRIBUTE.md and development policy

We should adopt a development policy and rules, for example that nobody pushes to master and all changes go in via pull requests. This is common for open source projects. I will draft one as a starting point with the usual recommendations and policies.

What goes to coding style, we should probably enforce PEP-8 as is or with minor changes (e.g. longer line length).

As a first step, I protected the master branch in Github settings:

--regress switch

Enhancement request. While --check-destination-dir regresses a damaged archive, there is no official way to regress an undamaged archive. I have written rdiff-backup-regress.sh which does this but in an unofficial way. There are legitimate reasons why one might want to regress an undamaged archive (including wishing to reverse the bloating that can happen when some large data gets unintentionally included in a new backup session).

At its simplest this could be a --regress switch which does what --check-destination-dir does but without first requiring that there be more than one current_mirror file. If this doesn't break anything...

Remove superfluous scripts from library directory

Issue by ericzolf
Sunday Apr 14, 2019 at 09:42 GMT
Originally opened as ericzolf#4

---

$ grep -le '#!' rdiff_backup/*.py
rdiff_backup/compilec.py
rdiff_backup/librsync_memoryleak2.py
rdiff_backup/myrdiff.py
rdiff_backup/profiled_rdb.py

Found too many current_mirror incs doesn't say where

When displaying this error message the path to the culprit directory should be included.

'RORPath instance has no attribute 'path'' raised of class '<type 'exceptions.AttributeError'>'

Got this exception executing rdiff-backup --no-acls

Exception 'RORPath instance has no attribute 'path'' raised of class '<type 'exceptions.AttributeError'>':
  File "rdiff_backup\Main.pyc", line 304, in error_check_Main
  File "rdiff_backup\Main.pyc", line 324, in Main
  File "rdiff_backup\Main.pyc", line 280, in take_action
  File "rdiff_backup\Main.pyc", line 343, in Backup
  File "rdiff_backup\backup.pyc", line 51, in Mirror_and_increment
  File "rdiff_backup\backup.pyc", line 243, in patch_and_increment
  File "rdiff_backup\rorpiter.pyc", line 284, in __call__
  File "rdiff_backup\backup.pyc", line 718, in start_process

Traceback (most recent call last):
  File "rdiff-backup", line 30, in <module>
  File "rdiff_backup\Main.pyc", line 304, in error_check_Main
  File "rdiff_backup\Main.pyc", line 324, in Main
  File "rdiff_backup\Main.pyc", line 280, in take_action
  File "rdiff_backup\Main.pyc", line 343, in Backup
  File "rdiff_backup\backup.pyc", line 51, in Mirror_and_increment
  File "rdiff_backup\backup.pyc", line 243, in patch_and_increment
  File "rdiff_backup\rorpiter.pyc", line 284, in __call__
  File "rdiff_backup\backup.pyc", line 718, in start_process
AttributeError: RORPath instance has no attribute 'path'

I'm using rdiff-backup-1.2.8 on Windows 10.

Does rdiff-backup intelligently handle log file rotation?

Some of our log files are huge, but right now I'm just looking at a simple case:
/var/log/letsencrypt/letsencrypt.log.N

Each time letsencrypt runs, it renames each file to N+1.

In a typical day, this will happen twice, which means files will move from N to N+2.

I suspect (but haven't traced) that rdiff-backup sees this as brand new data for each file.

Thankfully, letsencrypt's logs amount to about 5MB. OTOH, I have some log files that could easily be GBs large, and I'd be really sad if rdiff-backup had to resend all of them...

macOS: '[Errno 1] Operation not permitted' linking PkgInfo

On macOS, it seems that Apple disallows hardlinking files inside .app directories.

This happens at https://github.com/sol1/rdiff-backup/blob/8ccc5a3b44c996ecd810f8d5d586d0da6435cc32/rdiff-backup/rdiff_backup/backup.py#L568

I'm not entirely certain why the temp file is hardlinked and not moved. Would it be possible to fall back onto a move/copy?

What to do about bugs filed at Savannah?

There are a lot of open bugs at http://savannah.nongnu.org/bugs/?group=rdiff-backup

We should think about what to do with them. This is not a priority (after all they have been sitting there for years) but we should at some point deal with them. Maybe ask reporters to re-test with latest rdiff-backup 2.0 and re-report on Github if the issue still persists and they want it fixed.

Regression of failed backup with long file names can result in assertion error

This bug was originally reported here:
http://lists.nongnu.org/archive/html/rdiff-backup-users/2016-08/msg00000.html

The bug can be reproduced with the following script:

#!/bin/bash -v
# Create a long file name -- 211 characters.  This length is chosen to be less
# than the maximum allowed for ext4 filesystems (255 max.), but long enough for
# rdiff-backup to give it special treament (see longname.py).
longName=b
a=0123456789
for (( i = 0; i < 21; i++ )); do
longName=$longName$a
done

# Set up a source directory containing a file with the long name:
mkdir src
echo test1 > src/$longName
ls -l src

# Make a backup:
rdiff-backup src dst
sleep 1

# Keep a copy of the current_mirror file for use later:
cp dst/rdiff-backup-data/current_mirror* .

# Modify the src file:
echo test2 > src/$longName

# Make a 2nd backup:
rdiff-backup src dst

# Notice that the increment file is put in
# dst/rdiff-backup-data/long_filename_data/:
ls -l dst/rdiff-backup-data/long_filename_data

# Copy the saved current_mirror back so as to force rdiff-backup into
# concluding that the last backup failed  (a simulated failure for this test):
cp current_mirror* dst/rdiff-backup-data

# rdiff-backup will now report that there is a problem:
rdiff-backup --list-increments dst

# Perform the usual fix for the problem (regress the repository):
rdiff-backup --check-destination-dir dst

# See that rdiff-backup appears to be happy:
rdiff-backup --list-increments dst

# Here's the problem: regressing the repository failed to remove the increment
# file from the 2nd backup:
ls -l dst/rdiff-backup-data/long_filename_data

# Retry the 2nd backup.  It fails because the old increment file is in the way:
rdiff-backup src dst
rdiff-backup --list-increments dst

# Again, regress the repository to fix the failed backup:
rdiff-backup --check-destination-dir dst
rdiff-backup --list-increments dst

# This time, manually remove the increment file that should not be there (normally, this must
# be done more judiciously than illustrated here):
ls -l dst/rdiff-backup-data/long_filename_data
rm dst/rdiff-backup-data/long_filename_data/1*

# Now we are able to successfully make the 2nd backup:
rdiff-backup src dst
rdiff-backup --list-increments dst

As noted in the script, the problem is that regressing a backup using --check-destination-dir fails to completely clean up the repository when there are increment files stored under rdiff-backup-data/long_filename_data/.

A work-around for the bug is to use --check-destination-dir and then remove the problematic files under rdiff-backup-data/long_filename_data/. This removal needs to be done judiciously, however. The only files removed are ones where the date in the file name matches the date in the current_mirror filename. Here's an example where there is a match:

rdiff-backup-data/current_mirror.2016-08-20T21:30:01-04:00.data
rdiff-backup-data/long_filename_data/1.2016-08-20T21:30:01-04:00.diff.gz

Here is the script output when run:

#!/bin/bash -v
# Create a long file name -- 211 characters.  This length is chosen to be less
# than the maximum allowed for ext4 filesystems (255 max.), but long enough for
# rdiff-backup to give it special treament (see longname.py).
longName=b
a=0123456789
for (( i = 0; i < 21; i++ )); do
longName=$longName$a
done

# Set up a source directory containing a file with the long name:
mkdir src
echo test1 > src/$longName
ls -l src
total 8
-rw-rw-r--  1 joe joe 6 Aug 20 21:17 b012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789

# Make a backup:
rdiff-backup src dst
sleep 1

# Keep a copy of the current_mirror file for use later:
cp dst/rdiff-backup-data/current_mirror* .

# Modify the src file:
echo test2 > src/$longName

# Make a 2nd backup:
rdiff-backup src dst

# Notice that the increment file is put in
# dst/rdiff-backup-data/long_filename_data/:
ls -l dst/rdiff-backup-data/long_filename_data
total 16
-rw-rw-r--  1 joe joe 107 Aug 20 21:17 1.2016-08-20T21:17:56-04:00.diff.gz
-rw-------  1 joe joe   1 Aug 20 21:17 next_free

# Copy the saved current_mirror back so as to force rdiff-backup into
# concluding that the last backup failed (a simulated failure for this test):
cp current_mirror* dst/rdiff-backup-data

# rdiff-backup will now report that there is a problem:
rdiff-backup --list-increments dst
Fatal Error: Previous backup to /lan1/sys/home/joe/tmprdifftest/dst seems to have failed.
Rerun rdiff-backup with --check-destination-dir option to revert directory to state before unsuccessful session.

# Perform the usual fix for the problem (regress the repository):
rdiff-backup --check-destination-dir dst

# See that rdiff-backup appears to be happy:
rdiff-backup --list-increments dst
Found 0 increments:
Current mirror: Sat Aug 20 21:17:56 2016

# Here's the problem: regressing the repository failed to remove the increment
# file from the 2nd backup:
ls -l dst/rdiff-backup-data/long_filename_data
total 16
-rw-rw-r--  1 joe joe 107 Aug 20 21:17 1.2016-08-20T21:17:56-04:00.diff.gz
-rw-------  1 joe joe   1 Aug 20 21:17 next_free

# Retry the 2nd backup.  It fails because the old increment file is in the way:
rdiff-backup src dst
Exception 'Path: dst/rdiff-backup-data/long_filename_data/1.2016-08-20T21:17:56-04:00.diff.gz
Index: ('long_filename_data', '1.2016-08-20T21:17:56-04:00.diff.gz')
Data: {'uid': 500, 'perms': 436, 'type': 'reg', 'gname': 'joe', 'ea': <rdiff_backup.eas_acls.ExtendedAttributes instance at 0xb7c2b4cc>, 'ctime': 1471742278, 'devloc': 64770L, 'uname': 'joe', 'nlink': 1, 'gid': 500, 'mtime': 1471742276, 'atime': 1471742279, 'inode': 9043973L, 'size': 107L}' raised of class 'exceptions.AssertionError':
  File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 43, in Increment
    incrp = makediff(new, mirror, incpref)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 79, in makediff
    if compress: diff = get_inc(incpref, "diff.gz")
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 123, in get_inc
    assert not incrp.lstat(), incrp

Exception 'Path: dst/rdiff-backup-data/long_filename_data/1.2016-08-20T21:17:56-04:00.diff.gz
Index: ('long_filename_data', '1.2016-08-20T21:17:56-04:00.diff.gz')
Data: {'uid': 500, 'perms': 436, 'type': 'reg', 'gname': 'joe', 'ea': <rdiff_backup.eas_acls.ExtendedAttributes instance at 0xb7c2b4cc>, 'ctime': 1471742278, 'devloc': 64770L, 'uname': 'joe', 'nlink': 1, 'gid': 500, 'mtime': 1471742276, 'atime': 1471742279, 'inode': 9043973L, 'size': 107L}' raised of class 'exceptions.AssertionError':
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 698, in fast_process
    increment.Increment, (tf, mirror_rp, inc_prefix))
  File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 43, in Increment
    incrp = makediff(new, mirror, incpref)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 79, in makediff
    if compress: diff = get_inc(incpref, "diff.gz")
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 123, in get_inc
    assert not incrp.lstat(), incrp

Traceback (most recent call last):
  File "/usr/bin/rdiff-backup", line 30, in ?
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/backup.py", line 698, in fast_process
    increment.Increment, (tf, mirror_rp, inc_prefix))
  File "/usr/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 43, in Increment
    incrp = makediff(new, mirror, incpref)
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 79, in makediff
    if compress: diff = get_inc(incpref, "diff.gz")
  File "/usr/lib/python2.3/site-packages/rdiff_backup/increment.py", line 123, in get_inc
    assert not incrp.lstat(), incrp
AssertionError: Path: dst/rdiff-backup-data/long_filename_data/1.2016-08-20T21:17:56-04:00.diff.gz
Index: ('long_filename_data', '1.2016-08-20T21:17:56-04:00.diff.gz')
Data: {'uid': 500, 'perms': 436, 'type': 'reg', 'gname': 'joe', 'ea': <rdiff_backup.eas_acls.ExtendedAttributes instance at 0xb7c2b4cc>, 'ctime': 1471742278, 'devloc': 64770L, 'uname': 'joe', 'nlink': 1, 'gid': 500, 'mtime': 1471742276, 'atime': 1471742279, 'inode': 9043973L, 'size': 107L}
rdiff-backup --list-increments dst
Fatal Error: Previous backup to /lan1/sys/home/joe/tmprdifftest/dst seems to have failed.
Rerun rdiff-backup with --check-destination-dir option to revert directory to state before unsuccessful session.

# Again, regress the repository to fix the failed backup:
rdiff-backup --check-destination-dir dst
rdiff-backup --list-increments dst
Found 0 increments:
Current mirror: Sat Aug 20 21:17:56 2016

# This time, manually remove the increment file that should not be there (normally, this must
# be done more judiciously than illustrated here):
ls -l dst/rdiff-backup-data/long_filename_data
total 16
-rw-rw-r--  1 joe joe 107 Aug 20 21:17 1.2016-08-20T21:17:56-04:00.diff.gz
-rw-------  1 joe joe   1 Aug 20 21:17 next_free
rm dst/rdiff-backup-data/long_filename_data/1*

# Now we are able to successfully make the 2nd backup:
rdiff-backup src dst
rdiff-backup --list-increments dst
Found 1 increments:
    increments.2016-08-20T21:17:56-04:00.dir   Sat Aug 20 21:17:56 2016
Current mirror: Sat Aug 20 21:18:00 2016

OverflowError: group id is less than minimum

Can someone explain what is happening here?

I am running rdiff-backup version 1.2.8 on the latest build of raspbian on a RPi.
I want to back up some personal folders on a Synology NAS to a HDD attached to the RPi.
Currently I am setting up the system, so this is the first back up. For two folders it works fine. But on my folder with photos I get this error message.

According to the log-file the exception is thrown somewhere in the middle of the action. It seems that the process first continues, but soon after the process is stopped.

I just don’t understand the error. Is something wrong with the source folder?

This is the last part of the log-file (-v3):

Processing changed file Zeeland (2228-08-2004)/PICT0053.JPG
Processing changed file Zeeland (2228-08-2004)/Thumbs.db
Processing changed file Zwitserland (1527-07-2013)
Processing changed file Zwitserland (1527-07-2013)/@eadir
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09230.JPG
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09230.JPG/SYNOPHOTO_THUMB_B.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09230.JPG/SYNOPHOTO_THUMB_M.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09230.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg
Processing changed file Zwitserland (15~27-07-2013)/@eaDir/DSC09230.JPG/SYNOPHOTException 'group id is less than minimum' raised of class '<type 'exceptions.OverflowError'>':
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281, in call
last_branch.fast_process(*args)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 529, in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 559, in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 180, in copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 977, in chown
else: os.chown(self.path, uid, gid)

O_THUMB_S.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09230.JPG/SYNOPHOTO_THUMB_XL.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG/SYNOPHOTO_THUMB_B.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG/SYNOPHOTO_THUMB_M.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG/SYNOPHOTO_THUMB_S.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09231.JPG/SYNOPHOTO_THUMB_XL.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG/SYNOPHOTO_THUMB_B.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG/SYNOPHOTO_THUMB_M.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG/SYNOPHOTO_THUMB_S.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09232.JPG/SYNOPHOTO_THUMB_XL.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG/SYNOPHOTO_THUMB_B.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG/SYNOPHOTO_THUMB_M.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG/SYNOPHOTO_THUMB_PREVIEW.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG/SYNOPHOTO_THUMB_S.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09233.JPG/SYNOPHOTO_THUMB_XL.jpg
Processing changed file Zwitserland (1527-07-2013)/@eaDir/DSC09234.JPG
Processing changed file Zwitserland (15~27-07-2013)/@eaDir/DSC09234.JPG/SYNOPHOTO_THUMB_B.jpg
Traceback (most recent call last):
File "/usr/bin/rdiff-backup", line 30, in
rdiff_backup.Main.error_check_Main(sys.argv[1:])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
try: Main(arglist)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
take_action(rps)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
elif action == "backup": Backup(rps[0], rps[1])
File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 346, in Backup
backup.Mirror(rpin, rpout)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 38, in Mirror
DestS.patch(dest_rpath, source_diffiter)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 232, in patch
ITR(diff.index, diff)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281, in call
last_branch.fast_process(*args)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 529, in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 559, in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 180, in copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 977, in chown
else: os.chown(self.path, uid, gid)
OverflowError: group id is less than minimum