Fail when renaming with different case on case insensitive file system about rdiff-backup

[davekempe]

Hi there,
I know is is an annoying answer, but can you try 1.3.3? There are many fixes for this sort of problem in this version. It is very stable, despite the name.

[mkllnk]

Sure, almost no problem. But I can't compile it even though I have python2.7-dev and librsync-dev installed (Debian Wheezy). I tried it with python2.6 as well. No success.

python2.7 ./dist/setup.py build

running build
running build_py
copying rdiff_backup/SetConnections.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rpath.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/log.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/fs_abilities.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/eas_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/win_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/metadata.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/robust.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Main.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Globals.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/statistics.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/FilenameMapping.py -> build/lib.linux-x86_64-2.7/rdiff_backup
running build_ext
building 'rdiff_backup.C' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cmodule.c -o build/temp.linux-x86_64-2.7/cmodule.o
gcc: error: cmodule.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
error: command 'gcc' failed with exit status 4

[davekempe]

Hrmm, not sure why it isn't working for you. I built it from source on a fresh wheezy container just for testing:

root@wheezy-test:~# apt-get install librsync-dev python-dev wget
dev@wheezy-test:~$ wget http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-1.3.3.tar.gz
dev@wheezy-test:~$ tar zxf rdiff-backup-1.3.3.tar.gz
dev@wheezy-test:~/rdiff-backup-1.3.3$ python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/TempFile.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/log.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/SetConnections.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Globals.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/longname.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/static.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Hardlink.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/metadata.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rpath.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/librsync.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/fs_abilities.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/__init__.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/hash.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Main.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/user_group.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/rorpiter.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/eas_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Rdiff.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Time.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/restore.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/FilenameMapping.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/lazy.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/compare.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/statistics.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/win_acls.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/selection.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/backup.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/iterfile.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/manage.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/regress.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/robust.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/Security.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/increment.py -> build/lib.linux-x86_64-2.7/rdiff_backup
copying rdiff_backup/connection.py -> build/lib.linux-x86_64-2.7/rdiff_backup
running build_ext
building 'rdiff_backup.C' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cmodule.c -o build/temp.linux-x86_64-2.7/cmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/cmodule.o -o build/lib.linux-x86_64-2.7/rdiff_backup/C.so
building 'rdiff_backup._librsync' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c _librsyncmodule.c -o build/temp.linux-x86_64-2.7/_librsyncmodule.o
gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/_librsyncmodule.o -lrsync -o build/lib.linux-x86_64-2.7/rdiff_backup/_librsync.so
running build_scripts
creating build/scripts-2.7
copying and adjusting rdiff-backup -> build/scripts-2.7
copying and adjusting rdiff-backup-statistics -> build/scripts-2.7
changing mode of build/scripts-2.7/rdiff-backup from 644 to 755
changing mode of build/scripts-2.7/rdiff-backup-statistics from 644 to 755
dev@wheezy-test:~/rdiff-backup-1.3.3$ which python
/usr/bin/python
dev@wheezy-test:~/rdiff-backup-1.3.3$ ls -al /usr/bin/python
lrwxrwxrwx 1 root root 9 Sep 29  2013 /usr/bin/python -> python2.7
dev@wheezy-test:~/rdiff-backup-1.3.3$

I don't know what you did differently, but it does work building from source on wheezy.

[mkllnk]

Thank you. That worked. I tried to use the code from the github repository. The directory structure is different and setup.py is in the dist folder.

Anyway, I was able to build 1.3.3 and ran my script again on a FAT32 usb stick. Here is the error:

Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.AOP5zanMyu/rdiff-backup-data
Warning: hard linking not supported by filesystem at /media/C808-E853/tmp.AOP5zanMyu/rdiff-backup-data
Exception '[Errno 17] File exists: '/media/C808-E853/tmp.AOP5zanMyu/Archive'' raised of class '<type 'exceptions.OSError'>':
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)

Traceback (most recent call last):
  File "./rdiff-backup", line 30, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main
    try: Main(arglist)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 343, in Backup
    backup.Mirror_and_increment(rpin, rpout, incdir)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 51, in Mirror_and_increment
    DestS.patch_and_increment(dest_rpath, source_diffiter, inc_rpath)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 243, in patch_and_increment
    ITR(diff.index, diff)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 284, in __call__
    branch.start_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 725, in start_process
    self.prepare_dir(diff_rorp, self.base_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/backup.py", line 654, in prepare_dir
    base_rp.mkdir()
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 983, in mkdir
    self.conn.os.mkdir(self.path)
OSError: [Errno 17] File exists: '/media/C808-E853/tmp.AOP5zanMyu/Archive'

[LeonardMichlmayr]

I have just noticed some trouble with rdiff-backup on vfat in linux. The filesystem is not completely case-insensitive when mounted with the option iocharset=utf8.
$ touch A ; touch a
will result in an error, but
$ touch a; touch A
will not.

$ stat a
will not recognize the file A, but
$ touch a
is not allowed to create the file a if A exists.

rdiff-backup should always assume case insensitive behavior on vfat! If not, you will run into above error, as soon as there are two files with the same lowercase representation. Currently, rdiff-backup mistakes vfat as case sensitive!

[ericzolf]

OK, quick check on an USB stick with vfat and the result is... baaaad:

$ PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7 PATH=$PWD/build/scripts-3.7:$PATH rdiff-backup ../rdiff-backup_testfiles/various_file_types /run/media/myuser/MYUSBSTICK/backup1
Warning: hard linking not supported by filesystem at /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup-data
b'SpecialFileError fifo [Errno 1] Operation not permitted'
b"SpecialFileError symbolic_link [Errno 1] Operation not permitted: b'regular_file' -> b'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.8'"
OSError while renaming /run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13 to /run/media/myuser/MYUSBSTICK/backup1/ث�;087b��]��;021v;042�;015!�;062�;089����p��;019k;029���e�;085��;085;086���4��;088;003�;007sΞ��4;004�;023 ����جڅ�;075v;067�#���ɷ�_;015�g�;066;017;060=^�;077;019�c��;124;042;034;092'^$@#!(){};063+ ~`;032
b'UpdateError rdiff-backup.tmp.13 [Errno 22] Invalid argument: b\'/run/media/myuser/MYUSBSTICK/backup1/rdiff-backup.tmp.13\' -> b"/run/media/myuser/MYUSBSTICK/backup1/\\xd8\\xab\\xb1;087b\\xae\\xc5]\\x8a\\xbb;021v;042\\xf4;015!\\xf9;062\\xe2;089\\x86\\xbb\\xab\\xdbp\\xb0\\x84;019k;029\\xc2\\xf1\\xf5e\\xa5;085\\x82\\x9a;085;086\\xa0\\xf4\\xdf4\\xba\\xfd;088;003\\x82;007s\\xce\\x9e\\x8b\\xb34;004\\x9f;023 \\xf4\\x8f\\xa6\\xfa\\x97\\xab\\xd8\\xac\\xda\\x85\\xdc;075v;067\\xfa#\\x94\\x92\\x9e\\xc9\\xb7\\xc3_;015\\x84g\\x9a;066;017;060=^\\xdb;077;019\\x96c\\x8b\\xa7;124;042;034;092\'^$@#!(){};063+ ~`;032"'

After a sudo mkfs.ntfs -L MYUSBSTICK /dev/sda1, 2nd try on NTFS doesn't look better:

$ PYTHONPATH=$PWD/build/lib.linux-x86_64-3.7 PATH=$PWD/build/scripts-3.7:$PATH rdiff-backup ../rdiff-backup_testfiles/various_file_types /run/media/myuser/MYUSBSTICK/backup2
b"UpdateError two_hardlinked_files2 Updated mirror temp file b'/run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.11' does not match source"
OSError while renaming /run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.12 to /run/media/myuser/MYUSBSTICK/backup2/Ø«±Wb®Å]»;021v;042ô;015!ù;062âY»«Ûp°
                                         ;019k;029Âñõe¥UV ôß4ºýX;003;007sγ4;004¦ú«Ø¬Ú
ÜKvCú#
      g;017;060=^ÛM;019c§;124;042;034;092'^$@#!(){};063+ ~` 
b'UpdateError rdiff-backup.tmp.12 [Errno 22] Invalid argument: b\'/run/media/myuser/MYUSBSTICK/backup2/rdiff-backup.tmp.12\' -> b"/run/media/myuser/MYUSBSTICK/backup2/\\xc3\\x98\\xc2\\xab\\xc2\\xb1Wb\\xc2\\xae\\xc3\\x85]\\xc2\\x8a\\xc2\\xbb;021v;042\\xc3\\xb4;015!\\xc3\\xb9;062\\xc3\\xa2Y\\xc2\\x86\\xc2\\xbb\\xc2\\xab\\xc3\\x9bp\\xc2\\xb0\\xc2\\x84;019k;029\\xc3\\x82\\xc3\\xb1\\xc3\\xb5e\\xc2\\xa5U\\xc2\\x82\\xc2\\x9aUV\\xc2\\xa0\\xc3\\xb4\\xc3\\x9f4\\xc2\\xba\\xc3\\xbdX;003\\xc2\\x82;007s\\xc3\\x8e\\xc2\\x9e\\xc2\\x8b\\xc2\\xb34;004\\xc2\\x9f;023 \\xc3\\xb4\\xc2\\x8f\\xc2\\xa6\\xc3\\xba\\xc2\\x97\\xc2\\xab\\xc3\\x98\\xc2\\xac\\xc3\\x9a\\xc2\\x85\\xc3\\x9cKvC\\xc3\\xba#\\xc2\\x94\\xc2\\x92\\xc2\\x9e\\xc3\\x89\\xc2\\xb7\\xc3\\x83_;015\\xc2\\x84g\\xc2\\x9aB;017;060=^\\xc3\\x9bM;019\\xc2\\x96c\\xc2\\x8b\\xc2\\xa7;124;042;034;092\'^$@#!(){};063+ ~` "'
Exception '[Errno 84] Invalid or incomplete multibyte or wide character: b'/run/media/myuser/MYUSBSTICK/backup2/\xd8\xab\xb1Wb\xae\xc5]\x8a\xbb\x15v*\xf4\x0f!\xf9>\xe2Y\x86\xbb\xab\xdbp\xb0\x84\x13k\x1d\xc2\xf1\xf5e\xa5U\x82\x9aUV\xa0\xf4\xdf4\xba\xfdX\x03\x82\x07s\xce\x9e\x8b\xb34\x04\x9f\x17 \xf4\x8f\xa6\xfa\x97\xab\xd8\xac\xda\x85\xdcKvC\xfa#\x94\x92\x9e\xc9\xb7\xc3_\x0f\x84g\x9aB\x11<=^\xdbM\x13\x96c\x8b\xa7|*"\\\'^$@#!(){}?+ ~` '' raised of class '<class 'OSError'>':
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 384, in error_check_Main
    Main(arglist)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 406, in Main
    take_action(rps)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 342, in take_action
    Backup(rps[0], rps[1])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 431, in Backup
    backup.Mirror(rpin, rpout)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 39, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 264, in patch
    ITR(diff.index, diff)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rorpiter.py", line 316, in __call__
    last_branch.fast_process(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 577, in fast_process
    self.CCPP.get_rorps(index), self.basis_root_rp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 231, in get_mirror_inc_rps
    mirror_rp, alt_mirror, alt_inc = mir_triple_new(new_rorp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 195, in mir_triple_new
    mirror_rp = check_new_index(mirror_root, new_rorp.index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 140, in check_new_index
    rp = wrap_call(base.new_index, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 126, in wrap_call
    result = func(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1378, in new_index
    return self.__class__(self.conn, self.base, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/FilenameMapping.py", line 147, in __init__
    super().__init__(connection, base, index, data)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1041, in __init__
    if data is None: self.setdata()
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1067, in setdata
    self.data = self.conn.rpath.make_file_dict(self.path)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 355, in make_file_dict
    statblock = os.lstat(filename)

Traceback (most recent call last):
  File "/home/myuser/Public/rdiff-backup/build/scripts-3.7/rdiff-backup", line 32, in <module>
    rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 384, in error_check_Main
    Main(arglist)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 406, in Main
    take_action(rps)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 342, in take_action
    Backup(rps[0], rps[1])
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/Main.py", line 431, in Backup
    backup.Mirror(rpin, rpout)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 39, in Mirror
    DestS.patch(dest_rpath, source_diffiter)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 264, in patch
    ITR(diff.index, diff)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rorpiter.py", line 316, in __call__
    last_branch.fast_process(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/backup.py", line 577, in fast_process
    self.CCPP.get_rorps(index), self.basis_root_rp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 231, in get_mirror_inc_rps
    mirror_rp, alt_mirror, alt_inc = mir_triple_new(new_rorp)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 195, in mir_triple_new
    mirror_rp = check_new_index(mirror_root, new_rorp.index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 140, in check_new_index
    rp = wrap_call(base.new_index, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/longname.py", line 126, in wrap_call
    result = func(*args)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1378, in new_index
    return self.__class__(self.conn, self.base, index)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/FilenameMapping.py", line 147, in __init__
    super().__init__(connection, base, index, data)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1041, in __init__
    if data is None: self.setdata()
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 1067, in setdata
    self.data = self.conn.rpath.make_file_dict(self.path)
  File "/home/myuser/Public/rdiff-backup/build/lib.linux-x86_64-3.7/rdiff_backup/rpath.py", line 355, in make_file_dict
    statblock = os.lstat(filename)
OSError: [Errno 84] Invalid or incomplete multibyte or wide character: b'/run/media/myuser/MYUSBSTICK/backup2/\xd8\xab\xb1Wb\xae\xc5]\x8a\xbb\x15v*\xf4\x0f!\xf9>\xe2Y\x86\xbb\xab\xdbp\xb0\x84\x13k\x1d\xc2\xf1\xf5e\xa5U\x82\x9aUV\xa0\xf4\xdf4\xba\xfdX\x03\x82\x07s\xce\x9e\x8b\xb34\x04\x9f\x17 \xf4\x8f\xa6\xfa\x97\xab\xd8\xac\xda\x85\xdcKvC\xfa#\x94\x92\x9e\xc9\xb7\xc3_\x0f\x84g\x9aB\x11<=^\xdbM\x13\x96c\x8b\xa7|*"\\\'^$@#!(){}?+ ~` '

I don't try with exFAT but I expect a similar result. It doesn't look like rdiff-backup was ever meant to work from POSIX to non-POSIX conform file systems (it probably works more or less as long as no file is "strange"). It'll be quite an effort to make this kind of thing work reliably, don't hold your breath.

[ikus060]

Dod you test with master or with 1.2.8 ? Cause here, I'm using rdiff-backup 1.2.8 from Windows to Linux all the time. From NTFS or FAT32 to linux debian server.

I've never ran into issues.

[ikus060]

It make me think we should find a way to have integration test like this in our pipeline to start various backup with different filesystem and os for source and destination. I will keep this in mind and think about.

[ericzolf]

Windows to Linux is probably no issue, as it would be NTFS/FAT to ext4 (or other POSIX file system), the other way around is the issue here. Also, I thought that in general, the issue should be addressed together with longnames and other potential issues with filenames (like forbidden characters). Actually, there should be a central mapping function which derives a "valid and unique name for the target file system" for each filename it gets, either the filename itself, or if it's too long, wrong encoding, wrong characters, wrong type, etc... a replacement name with a mapping of some kind.

It's no simple task and requires some re-shuffling of the code, and probably a change in repository format, which I wouldn't like to address in the currently upcoming version. Actually I'd like first to simplify the code, remove some technical debt, before we go into implementing bigger functions. And as ikus060 highlighted, automated testing isn't a small feat either.

[ikus060]

Actually I'd like first to simplify the code, remove some technical debt, before we go into implementing bigger functions.

Completely agreed here. Let improve the current code base without adding too many feature. Let focus on have a robust release first.

[ollytom]

For comparison unison, when copying from a case-sensitive filesystem (e.g. Unix-like) to a case-insensitive one (e.g. NTFS on Windows or APFS on macOS), refuses to synchronise any files which have the same “spelling” but different case. A warning is printed and logged for each. See Case conflicts in “Cross-Platform Synchronization” in the unison documentation.

[ericzolf]

The current approach of rdiff-backup, despite bugs, is to map such names to unique valid names and restore the files based on the mapping, which I think is a good approach for a backup tool.

[ottok]

Hello @mklink!

The code base has been rewritten for Python 3 and there will be no further releases for the 1.x series in Python 2 anymore. Could you please help our efforts and test again with the latest development version (https://github.com/rdiff-backup/rdiff-backup#installing-the-latest-development-version) if you can still reproduce this and provide us with a test case (steps on how to reproduce the issue) that applies for latest development version?

With your help, if we can reproduce this problem still with the 2.x code base if might be fixed it in upcoming 2.x series releases.

[mkllnk]

Hi @ottok!

Thank you for following up. I just did the test again. The latest development version doesn't have instructions for Windows where I encountered the problem but your latest release seems to be up-to-date (7 days old). I downloaded rdiff-backup-1.9.1b0.win32exe.zip from the "3rd beta release before 2.0.0". Is that a 2.0 preview or how do I test this?

Anyway, the tested 1.9.1b0 still encounters the same error. Here is how I reproduced it:

mkdir s
mkdir d
mkdir d/test
touch d/test/a
./rdiff-backup.exe s d
mv s/test s/Test
./rdiff-backup.exe s d

I got the following output:

$ ./rdiff-backup.exe s d
Exception 'QuotedRPath: Path: 
d/rdiff-backup-data/increments/test.2020-03-01T09-49-59Z.dir
Index: ('test.2020-03-01T09-49-59Z.dir',)
Data: {'type': 'reg', 'size': 0, 'perms': 438, 'uid': 0, 'gid': 0, 
'inode': 4503599627465515, 'devloc': 1519922045, 'nlink': 1, 'mtime': 
1583055834, 'atime': 1583056225, 'ctime': 1583056225, 'uname': None, 
'gname': None, 'win_acl': b'# file: test.2020-03-01T09-49-59Z.dir
\nO:S-1-5-21-233083988-539155894-3873057244-1002G:S-1-5-21-233083988-539155894-3873057244-513D:(A;ID;FA;;;S-1-5-21-233083988-539155894-3873057244-1002)(A;ID;0x1200a9;;;S-1-5-21-233083988-539155894-3873057244-513)(A;ID;0x1200a9;;;WD)\n'}' 
raised of class '<class 'Assertio
nError'>':
   File "rdiff_backup\Main.py", line 390, in error_check_Main
Traceback (most recent call last):
   File "rdiff-backup", line 32, in <module>
   File "rdiff_backup\Main.py", line 390, in error_check_Main
   File "rdiff_backup\Main.py", line 412, in Main
   File "rdiff_backup\Main.py", line 348, in take_action
   File "rdiff_backup\Main.py", line 434, in Backup
   File "rdiff_backup\backup.py", line 55, in Mirror_and_increment
   File "rdiff_backup\backup.py", line 281, in patch_and_increment
   File "rdiff_backup\rorpiter.py", line 316, in __call__
   File "rdiff_backup\backup.py", line 804, in start_process
   File "rdiff_backup\increment.py", line 43, in Increment
   File "rdiff_backup\increment.py", line 118, in makedir
   File "rdiff_backup\increment.py", line 143, in get_inc
AssertionError: QuotedRPath: Path: 
d/rdiff-backup-data/increments/test.2020-03-01T09-49-59Z.dir
Index: ('test.2020-03-01T09-49-59Z.dir',)
Data: {'type': 'reg', 'size': 0, 'perms': 438, 'uid': 0, 'gid': 0, 
'inode': 4503599627465515, 'devloc': 1519922045, 'nlink': 1, 'mtime': 
1583055834, 'atime': 1583056225, 'ctime': 1583056225, 'uname': None, 
'gname': None, 'win_acl': b'# file: test.2020-03-01T09-49-59Z.dir
\nO:S-1-5-21-233083988-539155894-3873057244-1002G:S-1-5-21-233083988-539155894-3873057244-513D:(A;ID;FA;;;S-1-5-21-233083988-539155894-3873057244-1002)(A;ID;0x1200a9;;;S-1-5-21-233083988-539155894-3873057244-513)(A;ID;0x1200a9;;;WD)\n'}
[6380] Failed to execute script rdiff-backup

[mkllnk]

Hi @ottok!

I managed to replicate this issue on my Debian system. I created a FAT container and had the backup source and destination within that filesystem. Unfortunately, I didn't manage to run the latest master because of a signing key problem. Anyway, here is how to re-produce it with the old version:

cd /tmp/
fallocate -l 1m disk
/sbin/mkfs.vfat disk
udisksctl loop-setup -f disk
udisksctl mount -b /dev/loop0 # My system did that automatically.
cd /media/$USER/*/ # Select the right one if you have multiple mounts.
mkdir -p src/test
touch src/test/a
mkdir dst
rdiff-backup src dst
mv src/test/ src/Testt
mv src/Testt/ src/Test # You can't move it straight from `test` to `Test` because the FS reckons it's the same.
rdiff-backup src dst
# Exception 'QuotedPath: dst/rdiff-backup-data/increments/test.2020-03-05T09;05816;05839+11;05800.dir
# ...

[seihtam]

Hello I encountered the same error on Windows 10 using rdiff-backup 2.0.3. I encountered the same error as mkllnk by renaming a folder:

>mkdir from
>mkdir from\test
>mkdir to
>rdiff-backup.exe from to
>rename from\test Test
>rdiff-backup.exe from to
>rdiff-backup.exe from to
Exception 'QuotedRPath: Path: to/rdiff-backup-data/increments/test.2020-06-07T14-37-23+02-00.dir
Index: ('test.2020-06-07T14-37-23+02-00.dir',)
Data: {'type': 'reg', 'size': 0, 'perms': 438, 'uid': 0, 'gid': 0, 'inode': 26458647811148361, 'devloc': 2118428268, 'nlink': 1, 'mtime': 1591533431, 'atime': 1591533501, 'ctime': 1591533501, 'uname': None, 'gname': None, 'win_acl': b'# file: test.2020-06-07T14-37-23+02-00.dir\nO:BAG:S-1-5-21-2341865278-2280168835-870530268-513D:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;FA;;;S-1-5-21-2341865278-2280168835-870530268-1001)\n'}' raised of class '<class 'AssertionError'>':
  File "rdiff_backup\Main.py", line 393, in error_check_Main
  File "rdiff_backup\Main.py", line 415, in Main
  File "rdiff_backup\Main.py", line 351, in take_action
  File "rdiff_backup\Main.py", line 437, in Backup
  File "rdiff_backup\backup.py", line 55, in Mirror_and_increment
  File "rdiff_backup\backup.py", line 281, in patch_and_increment
  File "rdiff_backup\rorpiter.py", line 316, in __call__
  File "rdiff_backup\backup.py", line 820, in start_process
  File "rdiff_backup\increment.py", line 43, in Increment
  File "rdiff_backup\increment.py", line 118, in makedir
  File "rdiff_backup\increment.py", line 143, in get_inc

Traceback (most recent call last):
  File "rdiff-backup", line 32, in <module>
  File "rdiff_backup\Main.py", line 393, in error_check_Main
  File "rdiff_backup\Main.py", line 415, in Main
  File "rdiff_backup\Main.py", line 351, in take_action
  File "rdiff_backup\Main.py", line 437, in Backup
  File "rdiff_backup\backup.py", line 55, in Mirror_and_increment
  File "rdiff_backup\backup.py", line 281, in patch_and_increment
  File "rdiff_backup\rorpiter.py", line 316, in __call__
  File "rdiff_backup\backup.py", line 820, in start_process
  File "rdiff_backup\increment.py", line 43, in Increment
  File "rdiff_backup\increment.py", line 118, in makedir
  File "rdiff_backup\increment.py", line 143, in get_inc
AssertionError: QuotedRPath: Path: to/rdiff-backup-data/increments/test.2020-06-07T14-37-23+02-00.dir
Index: ('test.2020-06-07T14-37-23+02-00.dir',)
Data: {'type': 'reg', 'size': 0, 'perms': 438, 'uid': 0, 'gid': 0, 'inode': 26458647811148361, 'devloc': 2118428268, 'nlink': 1, 'mtime': 1591533431, 'atime': 1591533501, 'ctime': 1591533501, 'uname': None, 'gname': None, 'win_acl': b'# file: test.2020-06-07T14-37-23+02-00.dir\nO:BAG:S-1-5-21-2341865278-2280168835-870530268-513D:AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;FA;;;S-1-5-21-2341865278-2280168835-870530268-1001)\n'}
[12408] Failed to execute script rdiff-backup

I also tested the issue by renaming a file instead of a folder. This resulted in a different issue which can lead to data loss. Renaming a file to the same name with another case and backup up again will delete the file from the backup:

>mkdir from
>mkdir to
>echo 1>from/test.txt
>rdiff-backup.exe from to
>ren from\test.txt Test.txt
>rdiff-backup.exe from to
>rdiff-backup.exe --list-increments to
Found 1 increments:
    increments.2020-06-07T14-31-12+02-00.dir   Sun Jun  7 14:31:12 2020
Current mirror: Sun Jun  7 14:32:01 2020

>rdiff-backup --list-changed-since 5D to
changed .
new     Test.txt
deleted test.txt

>dir from
 Directory of C:\Users\Mat\Downloads\rdiff-backup-2.0.3\from

07-06-2020  14:31    <DIR>          .
07-06-2020  14:31    <DIR>          ..
07-06-2020  14:30                13 Test.txt
               1 File(s)             13 bytes
               2 Dir(s)  48.765.251.584 bytes free

>dir to
 Directory of C:\Users\Mat\Downloads\rdiff-backup-2.0.3\to

07-06-2020  14:31    <DIR>          .
07-06-2020  14:31    <DIR>          ..
07-06-2020  14:32    <DIR>          rdiff-backup-data
               0 File(s)              0 bytes
               3 Dir(s)  48.765.251.584 bytes free

[ericzolf]

Looking back into older issues. Here the problem is making a back-up from case-insensitive to case-insensitive file system. From case-sensitive to case-insensitive works just fine because rdiff-backup then quotes upper-case letters in names.

This brings us a workaround to this issue: force the quoting characters with something like rdiff-backup --override-chars-to-quote 'A-Z-"*/:<>?\\|;' src dst. This makes sure that upper-case characters are quoted, as well as other characters deemed not supported as well.

The actual solution to the issue is more complicated because it means comparing files in a case insensitive manner and I'm not sure at how many places exactly, so the proper solution will have to wait a bit more until the code has been cleaned-up. Hopefully, it'll then become clearer to me where to best do the changes.

[ericzolf]

Looking back at this bug, I don't think it'll ever get fixed:

1. it's actually talking about a case-insensitive file system where files are stored case sensitively, this is already rather sick.
2. and then the filename would change only by its case, still on a case-insensitive file system
3. and you decide to backup to yet another case insensitive file system
4. fixing this would have a performance impact on almost all use cases as it would add a new check for just this one very seldom (and rather sick) use case
5. if your backup fails on such an error then simply rename back the file and here you go, simple workaround...

I'm still open to arguments but I would rather document the workaround from my previous comment than spend time on this issue.

[ollytom]

Nice work @ericzolf. I think this makes sense. It looks like you and others have been very busy over the past couple of years! I'm not using rdiff-backup but I'd like to say thanks for all your hard work!