Running check_borg while running borg create fails about nagios-plugin-check_borg

[bebehei]

Yeah, I'd recommend scheduling this plugin right after your scheduled borg create should have finished.

Additionally I'd like to implement a check, which checks if the backup is locked and then bails properly with nagios check plugin output. And I guess one main point is the || exit ${STATUS_UNKNOWN} inside the subshell. This can't work.

nagios-plugin-check_borg/check_borg

Line 119 in 5137c68

last="$(${BORG} list --sort timestamp --last 1 --format '{time}' || exit ${STATE_UNKNOWN})"

[z3dm4n]

As a (quick) workaround for Failed to create/acquire the lock $BORG_REPO/lock.exclusive (timeout). I'll only check the service during workhours (predefined time period in icinga/nagios). Backup run is scheduled daily at 3:00. This should prevent interferences.

Because of the already mentioned limitation within borg's locking implementation aka borg list fails during an active run of borg create, I consider this workaround permanent until this behavior of borg gets changed in future releases.

[kosli]

I had a simliar issue with the lock file due to having the backups stored on mount point that is mounted read-only (see borgbackup/borg#2912 (comment)).
I expanded my wrapper script for borg which does remount-rw and remote-backup-borg so that it stores the json-stats of brog in a temp file, and upon finish of the borg command, it moves the file into its final location. -> and nagios check script reads out the values from the json file. Unfortunately the nagios php script is more a "proof of concept", but it is sufficient for my needs, can share somwhere if needed.

[z3dm4n]

@kosli
Nice idea. But I'd rather like to keep it simple - especially with backups. My approach is to do less aggressive checking, because the backup run is only once a day. Therefore the checks aren't as time critical as other service checks. Below my timeperiod definition for icinga1/nagios to circumvent the backup time. While the backup isn't running a normal_check_interval 60 seems to be reasonable.

# crontab -l
# m h  dom mon dow   command
0 3 * * * /usr/local/sbin/backup.sh >> /var/log/borg/backup.log 2>&1

define timeperiod{
        timeperiod_name 24x7-except-3-to-4am
        alias           24x7 except 3 to 4 AM
        sunday          00:00-03:00,04:00-24:00
        monday          00:00-03:00,04:00-24:00
        tuesday         00:00-03:00,04:00-24:00
        wednesday       00:00-03:00,04:00-24:00
        thursday        00:00-03:00,04:00-24:00
        friday          00:00-03:00,04:00-24:00
        saturday        00:00-03:00,04:00-24:00
}

[kosli]

@z3dm4n
Due to the read-only mounted backup media, I have to use the json approach, whereas I can keep that way also the json stats files for past backups, for reference, if I want. I thought I mention I approach here, even if it is just for reference/idea for others :-)