gkiefer / backup2l Goto Github PK
View Code? Open in Web Editor NEWbackup2l - A Low-Maintenance Backup/Restore Tool
Home Page: https://gkiefer.github.io/backup2l
License: GNU General Public License v2.0
backup2l - A Low-Maintenance Backup/Restore Tool
Home Page: https://gkiefer.github.io/backup2l
License: GNU General Public License v2.0
Backing up does not follow symbolic links as the default behaviour of 'find' is option -P. This might be unexpected to the average user (who realizes this only when the data is lost). I suggest to mention this in the man page and also in the explanations in backup2l.conf. Maybe also mention that option -L may be provided in SRCLIST to force find (and thus backup2l) follow symbolic links.
I've been using backup2l
for a long time and I love it. The files it generates are being backed up offsite to AWS Glacier. Glacier has no facility to move or rename the backups, or any of the associated metadata.
If I decide to purge some of the old full backups to save space (or if I reach the MAX_FULL
limit of 8), backup2l
will delete the first full backup and move/rename all the subsequent backups, moving them up. This move/rename is impossible to replicate on Glacier.
Is there a sensible workaround for this situation? Ideally backup2l
would never rename a BID once it's been created.
Hi everyone,
is this correct that backup2I modifies the "changed" timestamp from files? If so, is there a way to disable that (preserve)?.
Im just asking because i'm running openHAB and everytime a special file is changed, it get reread from oh and that triggered my home alarmsystem which is not quite good around 5 am every day ;-)
stat from one file - modify is ok, but change is bad in my case:
Access: 2017-04-01 07:44:29.068328380 +0200
Modify: 2017-04-01 07:44:28.884329360 +0200
Change: 2017-04-01 07:44:28.884329360 +0200
Greetings,
Hello Gundolf!
I am using your script and seems that summary information in mail reports is generated before archive is completely created.
Here is little piece of the mail report:
Wed Jul 18 05:42:18 BST 2018
Summary
=======
Backup Date Time | Size | Skipped Files+D | New Obs. | Err.
------------------------------------------------------------------------------
backup.1 2018-06-30 01:05 | 231.5G | 6 3764015 |3764015 0 | 0
backup.2 2018-07-09 01:06 | 113.7G | 7 3766091 |3766091 0 | 0
backup.21 2018-07-10 01:05 | 0.1G | 7 3766143 | 335 283 | 0
backup.22 2018-07-11 01:03 | 0.1G | 7 3766173 | 443 413 | 0
backup.23 2018-07-12 01:03 | 0.1G | 7 3766230 | 379 322 | 0
backup.24 2018-07-13 01:02 | 0.1G | 7 3766417 | 502 315 | 0
backup.25 2018-07-14 01:03 | 0.1G | 7 3766513 | 434 338 | 0
backup.26 2018-07-15 01:03 | 0.1G | 7 3766583 | 386 316 | 0
backup.27 2018-07-16 01:03 | 0.1G | 7 3766647 | 395 331 | 0
backup.3 2018-07-18 01:06 | 42.3G | 7 3766928 |3766928 0 | 0
while actual size for backup.3.tar.gz is 238039M (~238GiB).
Sizes for backup1 and backup2 looks correct so the issue happens just time to time.
It's not really big problem, but everytime I need to check backup filesizes manually to be sure that they was created succesfully.
I am using backup2l v1.6-pre2
Please, let me know if you need more details.
Thank you.
Hello Gundolf,
some time ago i wrote a dynamic Driver that is capable of automatically selecting pigz over gzip in case it is installed on the System. The patch proposed also changed the Default driver from gzip to pigz_or_gzip, so the backup is performed "out of the box" with the most performance available.
See my old proposal: https://sourceforge.net/p/backup2l/patches/20/
I think the patch does not apply cleanly anymore, but should be staightforward enough to easily be adjusted.
With best regards,
Beni
The option --no-recursion
should be placed before option -T
.
affected: DRIVER_TAR( )
, DRIVER_TAR_GZ( )
and DRIVER_TAR_BZ2( )
See: Debian Bug 829738#10
This happens using backup2l 1.5-8 from Kubuntu Xenial using the DRIVER_TAR_GZ_SPLIT driver:
klee@gonzales:/opt/restored$ sudo backup2l -r /
backup2l v1.5 by Gundolf Kiefer
Active files in <all.1011>: 368591
found in all.1011: 4613 (363978 left)
found in all.101: 41730 (322248 left)
found in all.1: 322248 ( 0 left)
Restoring 80513 directories...
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Restoring files...
I suggest providing to POST_BACKUP IDs of newly created and removed old backups.
It would be useful if encrypted backups are stored in a cloud. Here is the patch:
--- original-backup 2020-04-25 10:16:36.000000000 -0400
+++ patched-backup 2020-04-25 10:18:58.000000000 -0400
@@ -762,6 +762,7 @@
fi
# Remove old differential backups...
+ declare -a removed
if [[ $BASE_BID -gt 0 ]]; then
ARCH_LIST=(`ls $VOLNAME.*1.list.gz`)
LV=0
@@ -776,6 +777,7 @@
: $[MATCHCNT+=1]
if [[ $MATCHCNT -gt $GENERATIONS ]]; then
purge $BID
+ removed+=($BID)
fi
fi
done
@@ -824,7 +826,7 @@
# Run post-backup
echo
echo "Running post-backup procedure..."
- POST_BACKUP
+ POST_BACKUP $NEW_BID ${removed[@]}
echo
date
Hello
I'm trying to restore files from a pretty big archive file.
Unfortunately, even though all the files are present, i get this:
Active files in <all.1147>: 7192
found in all.1147: 0 ( 7192 left)
found in all.1146: 0 ( 7192 left)
found in all.1145: 0 ( 7192 left)
found in all.1144: 0 ( 7192 left)
found in all.1143: 0 ( 7192 left)
found in all.1142: 0 ( 7192 left)
found in all.1141: 0 ( 7192 left)
found in all.114: 0 ( 7192 left)
found in all.113: 0 ( 7192 left)
found in all.112: 0 ( 7192 left)
found in all.111: 1 ( 7191 left)
found in all.11: 0 ( 7191 left)
found in all.1: 7191 ( 0 left)
Cannot access archive file(s) of the following backup(s):
all.111
all.1
Nothing has been restored.
Am I missing something ?
Thanks for backup2l I like it a lot.
backup2l shows:
Creating archive using 'DRIVER_TAR_BZ2'...
tar: The following options were used after any non-optional arguments in archive create or update mode. These options are positional and affect only arguments that follow
+them. Please, rearrange them properly.
tar: --no-recursion has no effect
tar: Exiting with failure status due to previous errors
Everything seems to work fine.
What to do?
Greetings Gundolf,
I have been a big fan of backup2l for years. I use it on many of the Linux servers that I have administered over the past decade.
A few years ago, I was looking at some minor tweaks to contribute that would make backup2l work with MacOS and Red Hat / CentOS. I forked the repo and started tweaking and I realized that there were some basic portability issues that would best be addressed by abstracting away the name of the command-line tool (e.g., gsed instead of sed or gawk instead of awk) and moving some repeated code into new shell functions.
At some point I realized that I had added a lot of code to make the setup explicit, and also rewritten many lines to abstract away the command names. In other words, I had started to rewrite backup2l, retaining your style and code, mostly, but making it more portable to other *nix systems.
I also realized that if/when I finished, if I submitted a Pull Request of this magnitude, you would probably have a good laugh. It would be a lot of changes to review and the potential that something gets badly broken, no matter how carefully it is tested, is always there.
So, my question is whether I should continue in this direction and when finished and tested, submit a PR, or scrap this fork. It is not complete or tested, but it should be obvious from the diffs where I'm going. If you have a chance, take a look at the comparison : master...ferthalangur:dev_rbj-on-1.6
Thanks,
rob
Hi,
I am running backup2l via cygwin on Windows and it works well (I have to use the latest 1.6-pre2 for a problem in 1.5 that I now forget)
I had a situation recently where I was attempting to perform a restore onto a new laptop and the ownership and permissions basically failed badly. I presume this has more to do with cygwin's permission management than anything else.
For the content I was backing up, not having the permissions was not really problem - the content is most important.
For the actual files themselves I can override the DRIVER_TAR_GZ and remove the --same-permission --same-owner
arguments for tar
at :
Line 207 in 071ce97
But there is not anything I can do for the directories:
Lines 1012 to 1013 in 071ce97
I was thinking that if these 3 lines where extracted to a separate function, then that function could be overridden in the conf file if required.
thanks
Google Drive doesn’t support symlinks, so when I copy a set of backup2l-generated files to my Google Drive storage, the all.1.new.gz
file doesn’t get copied.
I suspect that other services such as Amazon Cloud Drive may also lack support for symlinks.
I use the rclone tool to copy backups to Google Drive and then to copy them back again, so the problem I have is, that all.1.new.gz
symlink doesn’t get round-tripped.
So I would like for backup2l to either just always create all.1.new.gz
by copying rather than symlinking, or to at least provide an option for doing that.
This happens using backup2l 1.5-8 from Kubuntu Xenial using the DRIVER_TAR_GZ_SPLIT driver:
klee@gonzales:/opt/restored$ sudo backup2l -r /
backup2l v1.5 by Gundolf Kiefer
Active files in <all.1011>: 368591
found in all.1011: 4613 (363978 left)
found in all.101: 41730 (322248 left)
found in all.1: 322248 ( 0 left)
Restoring 80513 directories...
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
Restoring files...
all.1011.tgz_split: 4613 file(s) using 'DRIVER_TAR_GZ_SPLIT'
all.101.tgz_split: 41730 file(s) using 'DRIVER_TAR_GZ_SPLIT'
tar: home/klee/.dropbox/command_socket: Not found in archive
tar: home/klee/.dropbox/iface_socket: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/All Mail: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/Important: Not found in archive
tar: home/klee/.thunderbird/axbuu1ii.default/ImapMail/imap.gmail.com/[Gmail].sbd/Starred: Not found in archive
tar: Exiting with failure status due to previous errors
Great tool!! I love it and use it on 6 raspberry pi. It is so great that I dare to comment on a minor thing, a misleading warning.
backup2l -s
can be run by a non-root user (and without sudo) as long as the backup directory is readable; the output is correct. However, there is a WARNING: Volume is or was locked by another instance of 'backup2l'., followed by instructions how to correct. I would have expected no warning or a warning that backup2l needs to be run by root. The correct hint for this warning is at the top of the backup2l output: /usr/sbin/backup2l: line 348: /var/backup/XXX.lock: Permission denied
what difference?
hey,
I use your backup scripton some machines, and also one with an InfluxDB and I want to backup the DB backup.
When you create a DB backup for InfluxDB it generated many files with the data because InfluxDB uses "shards" (data junks with all e.g. 7 days) internally you get one file per shard.But they all get a new date when they are generated. But older shard-backup-files that were not changed have the same data in, but a new timestamp.
Here it would be great to have the file content (means checksum) to be used to identify changes instead of the date o such.
Is there any chance to get this feature?
Filenames that contain linebreaks are not included in the backups:
$ ls mul*
'multiline'$'\n''filename'
$ ls mul* | hexdump -C
00000000 6d 75 6c 74 69 6c 69 6e 65 0a 66 69 6c 65 6e 61 |multiline.filena|
00000010 6d 65 0a |me.|
00000013
Output from backup2l:
...
Creating archive using 'DRIVER_TAR_GZ'...
tar: filename: Cannot stat: No such file or directory
tar: /home/xxx/xxx/xxx/multiline: Cannot stat: No such file or directory
...
< filename
< /home/xxx/xxx/xxx/multiline
It does not seem to be a limitation of tar (albeit the error message seems to imply), as tar can process this file:
$ tar --create --file /tmp/testtar.tar.gz ./multi*
$ tar --list --file /tmp/testtar.tar.gz
./multiline\nfilename
My system:
File with "[" or "]" in filename could not be restored
Driver = DRIVER_TAR
tar: path/Privat xy/\\[Untitled\\].pdf: Not found in archive
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.