gkiefer / backup2l Goto Github PK

Backing up does not follow symbolic links as the default behaviour of 'find' is option -P. This might be unexpected to the average user (who realizes this only when the data is lost). I suggest to mention this in the man page and also in the explanations in backup2l.conf. Maybe also mention that option -L may be provided in SRCLIST to force find (and thus backup2l) follow symbolic links.

I've been using backup2l for a long time and I love it. The files it generates are being backed up offsite to AWS Glacier. Glacier has no facility to move or rename the backups, or any of the associated metadata.

If I decide to purge some of the old full backups to save space (or if I reach the MAX_FULL limit of 8), backup2lwill delete the first full backup and move/rename all the subsequent backups, moving them up. This move/rename is impossible to replicate on Glacier.

Is there a sensible workaround for this situation? Ideally backup2l would never rename a BID once it's been created.

Hi everyone,

is this correct that backup2I modifies the "changed" timestamp from files? If so, is there a way to disable that (preserve)?.
Im just asking because i'm running openHAB and everytime a special file is changed, it get reread from oh and that triggered my home alarmsystem which is not quite good around 5 am every day ;-)

stat from one file - modify is ok, but change is bad in my case:

Access: 2017-04-01 07:44:29.068328380 +0200
Modify: 2017-04-01 07:44:28.884329360 +0200
Change: 2017-04-01 07:44:28.884329360 +0200

Greetings,

Hello Gundolf!

I am using your script and seems that summary information in mail reports is generated before archive is completely created.

Here is little piece of the mail report:

Wed Jul 18 05:42:18 BST 2018


Summary
=======

Backup       Date       Time  |  Size   | Skipped  Files+D |  New  Obs. | Err.
------------------------------------------------------------------------------
backup.1     2018-06-30 01:05 |  231.5G |       6  3764015 |3764015     0 |    0
backup.2     2018-07-09 01:06 |  113.7G |       7  3766091 |3766091     0 |    0
backup.21    2018-07-10 01:05 |    0.1G |       7  3766143 |  335   283 |    0
backup.22    2018-07-11 01:03 |    0.1G |       7  3766173 |  443   413 |    0
backup.23    2018-07-12 01:03 |    0.1G |       7  3766230 |  379   322 |    0
backup.24    2018-07-13 01:02 |    0.1G |       7  3766417 |  502   315 |    0
backup.25    2018-07-14 01:03 |    0.1G |       7  3766513 |  434   338 |    0
backup.26    2018-07-15 01:03 |    0.1G |       7  3766583 |  386   316 |    0
backup.27    2018-07-16 01:03 |    0.1G |       7  3766647 |  395   331 |    0
backup.3     2018-07-18 01:06 |   42.3G |       7  3766928 |3766928     0 |    0

while actual size for backup.3.tar.gz is 238039M (~238GiB).
Sizes for backup1 and backup2 looks correct so the issue happens just time to time.
It's not really big problem, but everytime I need to check backup filesizes manually to be sure that they was created succesfully.

I am using backup2l v1.6-pre2
Please, let me know if you need more details.
Thank you.

Hello Gundolf,
some time ago i wrote a dynamic Driver that is capable of automatically selecting pigz over gzip in case it is installed on the System. The patch proposed also changed the Default driver from gzip to pigz_or_gzip, so the backup is performed "out of the box" with the most performance available.

See my old proposal: https://sourceforge.net/p/backup2l/patches/20/
I think the patch does not apply cleanly anymore, but should be staightforward enough to easily be adjusted.

With best regards,
Beni

The option --no-recursion should be placed before option -T.

affected: DRIVER_TAR( ), DRIVER_TAR_GZ( ) and DRIVER_TAR_BZ2( )

See: Debian Bug 829738#10

This happens using backup2l 1.5-8 from Kubuntu Xenial using the DRIVER_TAR_GZ_SPLIT driver:

klee@gonzales:/opt/restored$ sudo backup2l -r /
backup2l v1.5 by Gundolf Kiefer

Active files in <all.1011>: 368591
found in all.1011: 4613 (363978 left)
found in all.101: 41730 (322248 left)
found in all.1: 322248 ( 0 left)

Restoring 80513 directories...
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Restoring files...

I suggest providing to POST_BACKUP IDs of newly created and removed old backups.
It would be useful if encrypted backups are stored in a cloud. Here is the patch:

--- original-backup	2020-04-25 10:16:36.000000000 -0400
+++ patched-backup	2020-04-25 10:18:58.000000000 -0400
@@ -762,6 +762,7 @@
     fi
 
     # Remove old differential backups...
+    declare -a removed
     if [[ $BASE_BID -gt 0 ]]; then
         ARCH_LIST=(`ls $VOLNAME.*1.list.gz`)
         LV=0
@@ -776,6 +777,7 @@
                     : $[MATCHCNT+=1]
                     if [[ $MATCHCNT -gt $GENERATIONS ]]; then
                         purge $BID
+			removed+=($BID)
                     fi
                 fi
             done
@@ -824,7 +826,7 @@
     # Run post-backup
     echo
     echo "Running post-backup procedure..."
-    POST_BACKUP
+    POST_BACKUP $NEW_BID ${removed[@]}
 
     echo
     date

Hello
I'm trying to restore files from a pretty big archive file.
Unfortunately, even though all the files are present, i get this:
Active files in <all.1147>: 7192
found in all.1147: 0 ( 7192 left)
found in all.1146: 0 ( 7192 left)
found in all.1145: 0 ( 7192 left)
found in all.1144: 0 ( 7192 left)
found in all.1143: 0 ( 7192 left)
found in all.1142: 0 ( 7192 left)
found in all.1141: 0 ( 7192 left)
found in all.114: 0 ( 7192 left)
found in all.113: 0 ( 7192 left)
found in all.112: 0 ( 7192 left)
found in all.111: 1 ( 7191 left)
found in all.11: 0 ( 7191 left)
found in all.1: 7191 ( 0 left)

Cannot access archive file(s) of the following backup(s):
all.111
all.1

Nothing has been restored.

Am I missing something ?

Thanks for backup2l I like it a lot.

backup2l shows:
Creating archive using 'DRIVER_TAR_BZ2'...
tar: The following options were used after any non-optional arguments in archive create or update mode. These options are positional and affect only arguments that follow
+them. Please, rearrange them properly.
tar: --no-recursion has no effect
tar: Exiting with failure status due to previous errors

Everything seems to work fine.
What to do?

Greetings Gundolf,

I have been a big fan of backup2l for years. I use it on many of the Linux servers that I have administered over the past decade.

A few years ago, I was looking at some minor tweaks to contribute that would make backup2l work with MacOS and Red Hat / CentOS. I forked the repo and started tweaking and I realized that there were some basic portability issues that would best be addressed by abstracting away the name of the command-line tool (e.g., gsed instead of sed or gawk instead of awk) and moving some repeated code into new shell functions.

At some point I realized that I had added a lot of code to make the setup explicit, and also rewritten many lines to abstract away the command names. In other words, I had started to rewrite backup2l, retaining your style and code, mostly, but making it more portable to other *nix systems.

I also realized that if/when I finished, if I submitted a Pull Request of this magnitude, you would probably have a good laugh. It would be a lot of changes to review and the potential that something gets badly broken, no matter how carefully it is tested, is always there.

So, my question is whether I should continue in this direction and when finished and tested, submit a PR, or scrap this fork. It is not complete or tested, but it should be obvious from the diffs where I'm going. If you have a chance, take a look at the comparison : master...ferthalangur:dev_rbj-on-1.6

Thanks,

rob

Hi,

I am running backup2l via cygwin on Windows and it works well (I have to use the latest 1.6-pre2 for a problem in 1.5 that I now forget)

I had a situation recently where I was attempting to perform a restore onto a new laptop and the ownership and permissions basically failed badly. I presume this has more to do with cygwin's permission management than anything else.

For the content I was backing up, not having the permissions was not really problem - the content is most important.

For the actual files themselves I can override the DRIVER_TAR_GZ and remove the --same-permission --same-owner arguments for tar at :

But there is not anything I can do for the directories:

I was thinking that if these 3 lines where extracted to a separate function, then that function could be overridden in the conf file if required.

thanks

Google Drive doesn’t support symlinks, so when I copy a set of backup2l-generated files to my Google Drive storage, the all.1.new.gz file doesn’t get copied.

I suspect that other services such as Amazon Cloud Drive may also lack support for symlinks.

I use the rclone tool to copy backups to Google Drive and then to copy them back again, so the problem I have is, that all.1.new.gz symlink doesn’t get round-tripped.

So I would like for backup2l to either just always create all.1.new.gz by copying rather than symlinking, or to at least provide an option for doing that.

This happens using backup2l 1.5-8 from Kubuntu Xenial using the DRIVER_TAR_GZ_SPLIT driver:

klee@gonzales:/opt/restored$ sudo backup2l -r /
backup2l v1.5 by Gundolf Kiefer

Active files in <all.1011>: 368591
found in all.1011: 4613 (363978 left)
found in all.101: 41730 (322248 left)
found in all.1: 322248 ( 0 left)

Restoring 80513 directories...
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Restoring files...
all.1011.tgz_split: 4613 file(s) using 'DRIVER_TAR_GZ_SPLIT'
all.101.tgz_split: 41730 file(s) using 'DRIVER_TAR_GZ_SPLIT'
tar: home/klee/.dropbox/command_socket: Not found in archive
tar: home/klee/.dropbox/iface_socket: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/All Mail: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/Important: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/Starred: Not found in archive
tar: Exiting with failure status due to previous errors

Great tool!! I love it and use it on 6 raspberry pi. It is so great that I dare to comment on a minor thing, a misleading warning.

backup2l -s can be run by a non-root user (and without sudo) as long as the backup directory is readable; the output is correct. However, there is a WARNING: Volume is or was locked by another instance of 'backup2l'., followed by instructions how to correct. I would have expected no warning or a warning that backup2l needs to be run by root. The correct hint for this warning is at the top of the backup2l output: /usr/sbin/backup2l: line 348: /var/backup/XXX.lock: Permission denied

what difference?

hey,

I use your backup scripton some machines, and also one with an InfluxDB and I want to backup the DB backup.
When you create a DB backup for InfluxDB it generated many files with the data because InfluxDB uses "shards" (data junks with all e.g. 7 days) internally you get one file per shard.But they all get a new date when they are generated. But older shard-backup-files that were not changed have the same data in, but a new timestamp.
Here it would be great to have the file content (means checksum) to be used to identify changes instead of the date o such.

Is there any chance to get this feature?

Filenames that contain linebreaks are not included in the backups:

$ ls mul*
'multiline'$'\n''filename'
$ ls mul* | hexdump -C
00000000  6d 75 6c 74 69 6c 69 6e  65 0a 66 69 6c 65 6e 61  |multiline.filena|
00000010  6d 65 0a                                          |me.|
00000013

Output from backup2l:

...
Creating archive using 'DRIVER_TAR_GZ'...
  tar: filename: Cannot stat: No such file or directory
  tar: /home/xxx/xxx/xxx/multiline: Cannot stat: No such file or directory
...
  < filename
  < /home/xxx/xxx/xxx/multiline

It does not seem to be a limitation of tar (albeit the error message seems to imply), as tar can process this file:

$ tar --create --file /tmp/testtar.tar.gz ./multi*
$ tar --list --file /tmp/testtar.tar.gz
./multiline\nfilename

My system:

• Linux xxx 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• backup2l v1.6 by Gundolf Kiefer
• stat (GNU coreutils) 8.30
• tar (GNU tar) 1.30

File with "[" or "]" in filename could not be restored
Driver = DRIVER_TAR
tar: path/Privat xy/\\[Untitled\\].pdf: Not found in archive