markfeit / duplicacy-scripts

A set of scripts for running Duplicacy on Unix systems

duplicacy-scripts

This is a set of scripts that can be used to manage running Duplicacy(GitHub) on Unix systems. They were developed and tested under Linux but should work in any environment that complies with POSIX.

NOTE: Limitations in the current implementation of Duplicacy and the fact that Windows does not handle symbolic linking in a POSIX-like way makes these scripts unsuitable for that environment.

Installation

To install this package on your system:

Clone this repository: git clone https://github.com/markfeit/duplicacy-scripts.git

cd duplicacy-scripts

Download the Duplicacy binary from Gilbert Chen's release page or build one of your own and and place it in the duplicacy-scripts directory as a file named duplicacy. Note that the license for Duplicacy imposes some restrictions on its use. Please abide by them.

Select a location where duplicacy-scripts, the Duplicacy configuration, its cache and log files will be kept. This location will be referred to as $DEST. The default is /opt/duplicacy.

Select the location which will form the root of the volume(s) to be backed up. This will be referred to as $ROOT. The default is /, which is suitable for most systems. (Specific parts of the filesystem may be included or excluded using Duplicacy's filter mechanism.)

Become root and execute:

• make install to install using the defaults
• make DEST=$DEST ROOT=$ROOT install to install using other directories.

Installation may be done as any other user, but be aware that this will limit the set of files backed up to those the user can read. In addition to installing these scripts, a .duplicacy file will be placed in $ROOT.

Set up Duplicacy by placing a preferences and optional filter file in $DEST/prefs Samples are provided in the prefs directory of the sources. These files are not installed by default.

Set up the scripts by editing $DEST/etc/settings. Note that if CONFIG_AUTO_UPDATE is enabled, any changes in the original GitHub repository will be applied to $DEST/etc/settings-update rather than overwriting settings.

At this point, backups and maintenance will be done automatically by cron.

Backups

Backups are run at 00:45 local time each morning. If there is another backup running (common when there is a long initial backup running), the newer backup will be aborted.

Logs of what happens during each backup and other matinenance activities are stored in $DEST/var/log. The latest logs or those for a specific date can be retrieved and read with $DEST/bin/logs.

Restoring Files

To restore files, execute $DEST/bin/restore. Detailed help may be obtained with the --help switch.

Maintenance

Daily (03:00):

• Prune old snapshots according to the rules in etc/prune.
• Remove old log files per CONFIG_LOG_LIFE in settings.
• Remove old cache files per CONFIG_CACHE_LIFE in settings.
• Pull and update the software (if enabled in $DEST/etc/settings).

Weekly (Sunday at 03:15):

• Fossilize and remove chunks that are no longer referenced.

• Check integrity. This verifies that all chunks that should be present and attempts to resurrect missing chunks from fossils if possible. There is no attempt to download and verify the contents of the chunks. (Not implemented yet.)

Monthly (First Sunday at 03:30):

• Nothing yet.

duplicacy-scripts's Issues

Explanation of preferences

Hello,

Could you explain the preferences ?

For instance: difference between id & name
what do no_backup, no_restore, no_save_password mean?

Thanks!

Full-Text Index of all files

Develop a system for indexing snapshot contents using a full-text indexer such as Xapian. Examples:

Find a specific file within a date range:

% bin/find-file local/home/bob/.bashrc date:2017..2018
Snapshot 2106  2018-05-12 02:23  local/home/bob/.bashrc
Snapshot 1255  2017-01-27 01:19  local/home/bob/.bashrc

Find files by text occurring in the name:

% bin/find-file .bashrc
Snapshot 2106  2018-05-12 02:23  local/home/bob/.bashrc
Snapshot 1957  2017-12-25 01:54  local/home/steve/.bashrc
Snapshot 1255  2017-01-27 01:19  local/home/bob/.bashrc

There will need to be three operations on the index:

• Initial build from all snapshot data
• Add data from new snapshot(s)
• Remove data for snapshots that no longer exist

Help with restore command

I'm trying to restore my repository to a DIR to test the backup integrity. But I'm struggling with the PATH variable.

If I want to restore the whole $ROOT backup as defined by the install command to another directory what should be command look like?

I've tried many iterations but the output is always the same:

root@nuc-smarthomehub .../duplicacy/bin# ./restore --dest /tmp/restore/ root 
Restoring from revision 1 as of 2021-10-05T22:37 from default
Restoring to /tmp/restore/
Storage set to gcd://Vault/nuc-smarthomehub-backups
Loaded 1 include/exclude pattern(s)
Forcing in-place mode with a non-default preference path
Parsing filter file /usr/bin/duplicacy/prefs/filters
Loaded 0 include/exclude pattern(s)
Restoring /tmp/restore to revision 1
Restored /tmp/restore to revision 1
Files: 0 total, 0 bytes
Downloaded 0 file, 0 bytes, 0 chunks
Skipped 0 file, 0 bytes
Total running time: 00:00:01

Thanks in advance!

Can we get a new release?

There have been some fixes in the master, which are not released yet. Are they still considered unstable?

If not, I would like to encourage you to create a new release containing them :)

REGEX errors

For some reason I cant restore my backup as it gives a regex error. it might be due to some silly thing that I am missing, but still cant get my head around it.

REGEX+=T(2[0-3]|[01][0-9]):[0-5][0-9]$: not found

this error keeps popping up when I use

$DIR/bin/restore --storage drive1 --time 2021-07-02T15:59 /

what is the mistake in this command?

$DIR/bin/restore --storage drive1 --list does return a list of backups so it should be possible to restore. any clues?

Install Issue 'cp: -r not specified; omitting directory 'duplicacy''

Sorry for being a complete novice.

I'm sure I'm following the instructions to the letter but I can't for the life of me get the make install command to complete successfully. Output below:

root@nuc-smarthomehub ~/duplicacy-scripts# make install                                        
rm -rf test test/root
find . -name "*~" | xargs rm -f
rm -f /opt/duplicacy/bin/duplicacy
cp -fp duplicacy /opt/duplicacy/bin/duplicacy
cp: -r not specified; omitting directory 'duplicacy'
make: *** [Makefile:68: /opt/duplicacy/bin/duplicacy] Error 1
root@nuc-smarthomehub ~/duplicacy-scripts# 

Within the github clone directory I did:

mkdir duplicacy and placed in there the downloaded binary duplicacy_linux_x64_2.7.2

Questions

Hi Mark,

Thanks for putting this together - very nice!

I have a few questions:

1. If I need to update a few of the actual commands (take backup for example) to include a different set of duplicacy options, will these be subsequently overwritten with an update?

2. Same question as 1) about cron jobs?

3. Is there any way to configure install/update to skip the cron job updates if you wanted to tailor them locally?

TIA!

Restore Commands

The restore command just gives OPTIONS and PATHS. What are the options and what are the paths? a bit of explanation would be nice. something like -help would also work. or is already there somewhere and I am missing it?

Filters

Right now root is the default for me which is /

Now I wanted to backup
/opt/folder1 and /home/user/folder2 only

I believe filter is what is required here. But when I use

+/opt/folder1
+/home/user/folder2

nothing is backed up, but when I use

+/opt
+/home

both folders are backed up. How do I specify only folder 1 and folder 2?

Update self-clobbers

When update is running and the script has been changed, the new version clobbers the old one mid-stream, which makes the shell unhappy.

The script should copy itself to a temporary location, exec the temporary copy and then remove it.