flyingcircusio / backy Goto Github PK

I just realize that we need to monitor for manual tags. We do have the global backy status check which ends up in Sensu, however, that's more about whether backy is creating new backups properly. Some way of reminding people that there are manual backups sitting around would be helpful. Let's extract that as a separate issue, though.

Originally posted by @ctheune in #14 (comment)

we could leverage rclone for the low level handling, however, our content-hashing system doesn't work with that directly, maybe we need completely different backup-type implementations but could leverage a centralized scheduler.

we regularly need to retire old servers. instead of copying data around we could just let an old server sit around for i.e. 3 months and start placing new backups on a new server. however, operator personnel may stumble over the issue of suddenly not finding older backups and having to hunt them down manually.

it would be nice if a backy status call would also show backups that are located on other backup servers (maybe even in remote locations?!?) so they can then go there to operate on them

and in this distributed setup we need to expire the old backups on the other servers and not just let them sit there infinitely.

this could also help rebalance backups between servers automatically.

Consider to automatically distrust backups in case of an error like:

Error: Header error - invalid compressed data

2021-09-17 06:09:26 [1349692] INFO Performing partial verification
2021-09-17 06:13:50 [1349692] INFO Removing old snapshot backy-RLCbehUaoQB3C8GbsLgXrF
2021-09-17 06:13:50 [1349692] ERROR Header error - invalid compressed data
Traceback (most recent call last):
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/main.py", line 334, in main
    func(**func_args)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/main.py", line 92, in backup
    b.backup(tags)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backup.py", line 59, in locked_function
    return f(self, *args, **kw)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backup.py", line 59, in locked_function
    return f(self, *args, **kw)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backup.py", line 187, in backup
    if not source.verify(backend):
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/sources/ceph/source.py", line 142, in verify
    return backy.utils.files_are_roughly_equal(source, target)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/utils.py", line 399, in files_are_roughly_equal
    chunk_b = b.read(blocksize)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backends/chunked/file.py", line 194, in read
    data, size = chunk.read(offset, size)
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backends/chunked/chunk.py", line 78, in read
    self._read_existing()
  File "/nix/store/1pn40rap4q6na5yvgq2sh2g2xkic4zc9-python3.8-backy-2.5.0dev/lib/python3.8/site-packages/backy/backends/chunked/chunk.py", line 61, in _read_existing
    data = lzo.decompress(data)
lzo.error: Header error - invalid compressed data

This is a follow up of https://fb.flyingcircus.io/f/cases/143474/

Instead of deleting the both files for a backup, it would be cool to have a ./backy forget <revision> command (or similar)

backy-extract handles this much more gracefully, the nbd server code is only vendored anyway and looks clunky.

• show what backups are available and SLA state (and then integrate that into the status page ticket creation process?)
• include S3 and VM images
• include offsite status

• bump python version (and dependencies)
• add annotations
• consider static checks (pre-commit?)
• demo IDE integration

When considering a separate job type for S3 backups vs. disk backups we could also create "replication" jobs that could be integrated as separate jobs that work on other jobs and that use the distributed scheduler API to place the remote backup somewhere else.

As most of that could be a simple rsync (maybe pay attention to extended attributes) we could even create automatic temporary SSH keys in a handshake process.

Alternatively, they could be features ontop of existing jobs with a unified API for marking which jobs want to be backed up to a remote destination.

This could either be an async block thing (I don't think so) or the jobs listing is accessing data where a file is locked and needs to wait until the backup is done (that's my guess).

• changelog entry
• impact

see fc-nixos templates