In CentOS 7 a different path is seemingly needed for the KVM variable, because:

[root@dualix /]# find / -type f -name "kvm"
...none found...
[root@dualix /]# find / -type f -name "qemu-kvm"
/usr/libexec/qemu-kvm

Therefore, the following add-on (after # Add this...) can help:

Executables

QEMU_IMG="/usr/bin/qemu-img"
VIRSH="/usr/bin/virsh"
KVM="/usr/bin/kvm"
if [ -x "/usr/bin/qemu-kvm" ]; then
KVM="/usr/bin/qemu-kvm"
fi

Add this! (SV, for CentOS 7)

if [ -x "/usr/libexec/qemu-kvm" ]; then
KVM="/usr/libexec/qemu-kvm"
fi

On my system:

[root@dualix /]# grep -i centos /etc/*
/etc/redhat-release:CentOS Linux release 7.1.1503 (Core)
/etc/system-release:CentOS Linux release 7.1.1503 (Core)

After this addition the fi-backup.sh script works on my CentOS 7. Maybe a case analysis, system-by-system is required?

I have been using fi-backup for several years, to manage fifty VMs, I find the product very reliable but I encountered an error today that I do not understand.

consolidation of a VM fails with the following message:

VIRSH : /usr/bin/virsh -q blockcommit themawebprod /var/lib/libvirt/images/themawebprod.bimg-20220130-011709 --wait --pivot --active
[ERR] Error consolidating block device '/var/lib/libvirt/images/themawebprod.bimg-20220130-011709' for 'themawebprod':
 error: No complete monitor response found in 10485760 bytes: Numerical result out of range

the last successful consolidation dates from January 26, 7:20 am. it contains the following 4 virtual disks:

root@kvm2540:~ # ls -lthr /var/lib/libvirt/images/themawebprod*
-rw------- 1 libvirt-qemu kvm  28G Jan 26 07:20 /var/lib/libvirt/images/themawebprod-2.qcow2
-rw------- 1 libvirt-qemu kvm  21G Jan 26 07:20 /var/lib/libvirt/images/themawebprod-3.qcow2
-rw------- 1 libvirt-qemu kvm  49G Jan 26 07:20 /var/lib/libvirt/images/themawebprod-1.qcow2
-rwxr-xr-x 1 libvirt-qemu kvm  18G Jan 26 07:20 /var/lib/libvirt/images/themawebprod.qcow2*

since then, hourly snapshots (bimg) have been piling up and daily consolidation has failed.

-rw------- 1 libvirt-qemu kvm 157M Jan 26 08:16 /var/lib/libvirt/images/themawebprod.bimg-20220126-072049
-rw------- 1 libvirt-qemu kvm 1.2M Jan 26 08:16 /var/lib/libvirt/images/themawebprod-1.bimg-20220126-072049
-rw------- 1 libvirt-qemu kvm 8.6M Jan 26 08:16 /var/lib/libvirt/images/themawebprod-3.bimg-20220126-072049
-rw------- 1 libvirt-qemu kvm 2.5M Jan 26 08:17 /var/lib/libvirt/images/themawebprod-2.bimg-20220126-072049
-rw------- 1 libvirt-qemu kvm 4.9M Jan 26 09:16 /var/lib/libvirt/images/themawebprod-2.bimg-20220126-081704
-rw------- 1 libvirt-qemu kvm 152M Jan 26 09:16 /var/lib/libvirt/images/themawebprod.bimg-20220126-081704
-rw------- 1 libvirt-qemu kvm 1.3M Jan 26 09:16 /var/lib/libvirt/images/themawebprod-1.bimg-20220126-081704
-rw------- 1 libvirt-qemu kvm 8.9M Jan 26 09:16 /var/lib/libvirt/images/themawebprod-3.bimg-20220126-081704
...
-rw------- 1 libvirt-qemu kvm 3.5M Jan 30 01:14 /var/lib/libvirt/images/themawebprod-2.bimg-20220130-001709
-rw------- 1 libvirt-qemu kvm 151M Jan 30 01:17 /var/lib/libvirt/images/themawebprod.bimg-20220130-001709
-rw------- 1 libvirt-qemu kvm 1.2M Jan 30 01:17 /var/lib/libvirt/images/themawebprod-1.bimg-20220130-001709
-rw------- 1 libvirt-qemu kvm 8.4M Jan 30 01:17 /var/lib/libvirt/images/themawebprod-3.bimg-20220130-001709
-rw------- 1 libvirt-qemu kvm  49M Jan 30 11:34 /var/lib/libvirt/images/themawebprod-2.bimg-20220130-011709
-rw------- 1 libvirt-qemu kvm 8.0M Jan 30 11:50 /var/lib/libvirt/images/themawebprod-1.bimg-20220130-011709
-rw------- 1 libvirt-qemu kvm 336M Jan 30 11:50 /var/lib/libvirt/images/themawebprod.bimg-20220130-011709
-rw------- 1 libvirt-qemu kvm  73M Jan 30 11:50 /var/lib/libvirt/images/themawebprod-3.bimg-20220130-011709

so what's going on? I don't know where to look...

Thanks in advance!

Ernest.

I read the nuts & bolts as well as the README and nowhere is it mentioned that the newer commits still include a backup of the libvirt XML definitions. Am I parsing the arguments wrong or is that functionality no longer present?

I've two different source directories where the filenames are the same.
This leads to overwritten files in case of copy to the backup directory during "Snapshot and consolidation" with a given backup directory. I will rename my source files two prevent those duplicates. Maybe it is a good idea to issue an error in this case.

Thank you for this piece of software!

Hi,

I am yet another happy user of your backup script. I did have a question about the backups. I presume this only backs up the disk image, right? Will we need another process to dump the domain's XML as part of a full backup? Or is that somehow in the image as well?

See https://help.github.com/articles/licensing-a-repository/.
Example of what I mean: https://github.com/sysown/proxysql/blob/v2.0.2/LICENSE

Benefit: Github detects the license correctly.

Very good job, Davide ! thanks.

but when LANG is set to fr_FR.UTF-8, fi_backup fails because the VM is not seen as running, as you can see here :

# LANG=fr_FR.UTF-8 && ./fi-backup.sh -c -b /var/lib/libvirt/backups/generic -d generic
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup:  generic
[DEB] Domains RUNNING to backup: 
[ERR] Consolidation only works with running domains. 'generic' is not running! Doing full backup only of 'generic'
[ERR] Skipping consolidation/backup of 'generic'

the problem comes from the result of the virsh command. it answers "en cours d'exécution" instead of "running".

# LANG=fr_FR.UTF-8 && virsh -q domstate generic
en cours d'exécution

# LANG=en_US.UTF-8 && virsh -q domstate generic
running

Consequently, i need to prefix the fi_backup command with LANG=en_US.UTF-8

# LANG=en_US.UTF-8 && ./fi-backup.sh -c -b /var/lib/libvirt/backups/generic -d generic
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  generic
[DEB] Consolidation of block devices for 'generic' requested
[DEB] Block devices to be consolidated:
	/var/lib/libvirt/images/generic.bimg-20180223-155100
[DEB] Consolidation method: blockcommit
[DEB] Consolidation of '0' block device: '/var/lib/libvirt/images/generic.bimg-20180223-155100' for 'generic'
[DEB] Parent block device: '/var/lib/libvirt/images/generic.qcow2'
[VER] Consolidation of block device '/var/lib/libvirt/images/generic.bimg-20180223-155100' for 'generic' successful
[DEB] Not deleting last element of snapshot_chain (top parent) since consolidation method='blockcommit'
[VER] Deleting old backing files for 'generic'
[VER] Deleting old backing file '/var/lib/libvirt/images/generic.bimg-20180223-155100' for 'generic'

what do you think ?
is it possible to fix the code perennially ?
Thanks in advance.

Ernest.

Consolidation failed this morning, with the following error: "block copy still active: disk 'sda' not ready for pivot yet".

# LANG=en_US.UTF-8 && /usr/local/bin/fi-backup.sh -c -b /var/lib/libvirt/backups/ubuntu  -d ubuntu
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  ubuntu
[DEB] Consolidation of block devices for 'ubuntu' requested
[DEB] Block devices to be consolidated:
/var/lib/libvirt/images/ubuntu.bimg-20180824-161701
[DEB] Consolidation method: blockcommit
[DEB] Consolidation of '0' block device: '/var/lib/libvirt/images/ubuntu.bimg-20180824-161701' for 'ubuntu'
[DEB] Parent block device: '/var/lib/libvirt/images/ubuntu.bimg-20180824-151702'
[ERR] Error consolidating block device '/var/lib/libvirt/images/ubuntu.bimg-20180824-161701' for 'ubuntu':
 error: failed to pivot job for disk /var/lib/libvirt/images/ubuntu.bimg-20180824-161701
error: block copy still active: disk 'sda' not ready for pivot yet

The libvirt images directory contained the following files:

# ls -ltr /var/lib/libvirt/images/ubuntu*
-rw------- 1 libvirt-qemu kvm  1179254784 Aug 24 14:54 /var/lib/libvirt/images/ubuntu.bimg-20180824-144024
-rw------- 1 libvirt-qemu kvm   232194048 Aug 24 15:15 /var/lib/libvirt/images/ubuntu.bimg-20180824-145422
-rw------- 1 libvirt-qemu kvm     5308416 Aug 24 16:17 /var/lib/libvirt/images/ubuntu.bimg-20180824-151702
-rw------- 1 libvirt-qemu kvm   361168896 Aug 27 11:50 /var/lib/libvirt/images/ubuntu.bimg-20180824-161701
-rw-r-xr-- 1 libvirt-qemu kvm 14377746432 Aug 27 11:50 /var/lib/libvirt/images/ubuntu.qcow2

I tried to see if there were some errors in ubuntu.bimg-* files, but everything seemed normal.

# virsh blockjob ubuntu /var/lib/libvirt/images/ubuntu.bimg-20180824-161701 --info
Active Block Commit: [100 %]

# qemu-img info /var/lib/libvirt/images/ubuntu.bimg-20180824-161701
image: /var/lib/libvirt/images/ubuntu.bimg-20180824-161701
file format: qcow2
virtual size: 16G (17179869184 bytes)
disk size: 344M
cluster_size: 65536
backing file: /var/lib/libvirt/images/ubuntu.bimg-20180824-151702 
backing file format: qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

I tried different things, but to no avail...

# virsh blockjob ubuntu sda --abort
# /usr/bin/virsh -q blockcommit ubuntu /var/lib/libvirt/images/ubuntu.bimg-20180824-161701 --wait --pivot --active
error: failed to pivot job for disk /var/lib/libvirt/images/ubuntu.bimg-20180824-161701
error: block copy still active: disk 'sda' not ready for pivot yet

Finally, as suggested here, i changed the virtual disk Cache mode from Hypervisor default to none... and then blockcommit completed successfully!

# LANG=en_US.UTF-8 && /usr/local/bin/fi-backup.sh -c -b /var/lib/libvirt/backups/ubuntu  -d ubuntu
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  ubuntu
[DEB] Consolidation of block devices for 'ubuntu' requested
[DEB] Block devices to be consolidated:
/var/lib/libvirt/images/ubuntu.bimg-20180824-161701
[DEB] Consolidation method: blockcommit
[DEB] Consolidation of '0' block device: '/var/lib/libvirt/images/ubuntu.bimg-20180824-161701' for 'ubuntu'
[DEB] Parent block device: '/var/lib/libvirt/images/ubuntu.bimg-20180824-151702'
[VER] Consolidation of block device '/var/lib/libvirt/images/ubuntu.bimg-20180824-161701' for 'ubuntu' successful
[DEB] Not deleting last element of snapshot_chain (top parent) since consolidation method='blockcommit'
[VER] Deleting old backing files for 'ubuntu'
[VER] Deleting old backing file '/var/lib/libvirt/images/ubuntu.bimg-20180824-161701' for 'ubuntu'
[VER] Deleting old backing file '/var/lib/libvirt/images/ubuntu.bimg-20180824-151702' for 'ubuntu'
[VER] Deleting old backing file '/var/lib/libvirt/images/ubuntu.bimg-20180824-145422' for 'ubuntu'
[VER] Deleting old backing file '/var/lib/libvirt/images/ubuntu.bimg-20180824-144024' for 'ubuntu'

What do you think?
The proposed solution is 3 years old... Is this bug still relevant?

Thanks in advance.

Consolidation was working perfectly with Ubuntu 16.04, but it fails with Ubuntu 18.04.
The problem is the new version, QEMU 2.10 comes with enabled builtin file locking to prevent image corruption, as explained here

The error message is:

[DEB] Getting all backing files for: '/var/lib/libvirt/images/salinois.bimg-20190728-115225'
qemu-img: Could not open '/var/lib/libvirt/images/salinois.bimg-20190728-115225': Failed to get shared "write" lock
Is another process using the image?

i do not see what i can do to solve the problem...

Thanks again for a nice program. If I've read the code right, I see that there is a rebasing of the backing files in the backup directory which stores the full path to that source in the backup directory.

So you currently have

   new_backing_file="$BACKUP_DIRECTORY/$(basename $backing_file)"
...
   new_parent_backing_file=$BACKUP_DIRECTORY/$(basename $parent_backing_file)
...
   command_output=$($QEMU_IMG rebase -u -b "$new_parent_backing_file" "$new_backing_file")

I think that would make sense if you ran the VM in the backup directory, but in the case of a disaster recovery (DR) you'd put the backup files back in their original location, not run from the backup directory (which might potentially cause an issue anyway as this script would then be backing up the VM from and to the same directory).

So I think it might make more sense to not rebase the backup files. What do you think?

Using eval ${....}= breaks finding files.

Reverting to previous code until a cleaner solution is found.

Consolidation fails with a Windows 2016 Server, i do not understand why... Backups are performed every hour, and consolidation once a day.

Below the logs:

[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  Magnus
[DEB] Snapshot for domain 'Magnus' requested
[DEB] Using timestamp '20180720-131701'
[DEB] Snapshotting block devices for 'Magnus' using suffix 'bimg-20180720-131701'
Formatting '/var/lib/libvirt/images/Magnus.bimg-20180720-131701', fmt=qcow2 size=42949672960 backing_file=/var/lib/libvirt/images/Magnus.bimg-20180720-123807 backing_fmt=qcow2 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
[VER] Snapshot for block devices of 'Magnus' successful
[DEB] Getting all backing files for: '/var/lib/libvirt/images/Magnus.bimg-20180720-131701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-123807' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-123807'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-121701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-121701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-111701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-111701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.bimg-20180720-101701' to '/var/lib/libvirt/backups/Magnus/Magnus.bimg-20180720-101701'
[VER] Copy backing file '/var/lib/libvirt/images/Magnus.qcow2' to '/var/lib/libvirt/backups/Magnus/Magnus.qcow2'
[INF] libVirt version '1.3.1' support is experimental
[DEB] /usr/bin/qemu-img version '2.5.0' is supported
[DEB] QEMU/KVM version '2.5.0' is supported
[DEB] Domains NOTRUNNING to backup: 
[DEB] Domains RUNNING to backup:  Magnus
[DEB] Consolidation of block devices for 'Magnus' requested
[DEB] Block devices to be consolidated:
    /var/lib/libvirt/images/Magnus.bimg-20180720-131701
[DEB] Consolidation method: blockcommit
[DEB] Consolidation of '0' block device: '/var/lib/libvirt/images/Magnus.bimg-20180720-131701' for 'Magnus'
[DEB] Parent block device: '/var/lib/libvirt/images/Magnus.bimg-20180720-123807'
[ERR] Error consolidating block device '/var/lib/libvirt/images/Magnus.bimg-20180720-131701' for 'Magnus':
 error: block copy still active: disk 'sda' already in active block job

What's the matter ?

I also use fi-backup with other Ubuntu VM, and it works very well. Would the problem be specific to Windows VM?

Can you help me, please ?

I created some live backups of a virtual machine and afterwards wanted
to consolidate them into a single image. I turned the VM off and used the
following command:

sudo ./fi-backup.sh -c -d t64web
[INF] libVirt version '1.2.8' support is experimental
[DEB] /usr/bin/qemu-img version '2.1.0' is supported
[DEB] QEMU/KVM version '2.1.0' is supported
[DEB] Domains NOTRUNNING:  t64web
[DEB] Domains RUNNING: 
[DEB] Backing up: cp -up  /
[DEB] Parent block device: '/var/lib/libvirt/images/t64web.bimg-20151029-083223'
[DEB] Backing up: cp -up /var/lib/libvirt/images/t64web.bimg-20151029-083223 /
[DEB] Next file in backing file chain: '/var/lib/libvirt/images/t64web.qcow2'
[DEB] Parent block device: '/var/lib/libvirt/images/t64web.qcow2'
[DEB] Backing up: cp -up /var/lib/libvirt/images/t64web.qcow2 /
[DEB] Next file in backing file chain: ''
[DEB] All 3 block files for 't64web': /var/lib/libvirt/images/t64web.bimg-20151029-083332 : /var/lib/libvirt/images/t64web.bimg-20151029-083223 /var/lib/libvirt/images/t64web.bimg-20151029-083223 /var/lib/libvirt/images/t64web.qcow2

I expected that consolidation for not running domains works the same way as for running domains. Nevertheless, that is currently not the case. Instead the script started to back up the image files to the root folder /. I had a quick tool into the script and it seems that it simply backups all
domains which are currently not running. I think this should be fixed.

I am using version 2.1.0 which I pulled from the master branch on Oct. 29th.

$ ./fi-backup.sh -V
fi-backup version 2.1.0

Hello,

i currently test this script first issue

[ERR] Unsupported libVirt version '1.2.9'. Please use libVirt 0.9.13 or greather

HTH

regards max

Any plans for using the latest features of libvirt? This method is way faster than the current one.

http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

It's a nice script operating flawlessly for other single-disk VMs on the same host.
However, something goes wrong for a two-disk VM I just added.

All other disk images are owned by root:root, as the images in question, so it's not a permission issue.

The fi-backup.sh script seems to erroneously report "error getting backing file",
although all the files exist and are OK, and the VM continues to operate fine.

There are two .qcow2 disks in the domain:
getafix-hda.*
getafix-hdb.*
which the script finds and snapshots correctly. But there is a (spurious) error message
and the he previous day's backing files are not transferred to the backup directory,
as they should and moved for my other single-disk VMs.

The command I use is:

./fi-backup.sh -b /var/lib/libvirt/images-backups -d $name

where $name cycles through all the machines' domain names I want to backup.
Again: it works smoothly for all single-disk VMs, but errs on the only two-disk VM.

===== 2015-05-25 03:27:02, backing up getafix.intertex.se =====
[DEB] libVirt version '1.2.8' is supported
[DEB] /usr/bin/qemu-img version '1.5.3' is supported
[DEB] KVM version '1.5.3' is supported
[DEB] Snapshot for domain 'getafix.intertex.se' requested
[DEB] Using timestamp '20150525-032702'
[DEB] Snapshotting block devices for 'getafix.intertex.se' using suffix 'bimg-20150525-032702'
[VER] Snapshot for block devices of 'getafix.intertex.se' successful
qemu-img: Could not open '/var/lib/libvirt/images/getafix-hda.bimg-20150525-032702
/var/lib/libvirt/images/getafix-hdb.bimg-20150525-032702': Could not open file: No such file or directory
[ERR] Error getting backing file for '/var/lib/libvirt/images/getafix-hda.bimg-20150525-032702
/var/lib/libvirt/images/getafix-hdb.bimg-20150525-032702'.

However:

[root@dualix images]# ll getafix-hda.*
-rw-------. 1 root root 4333371392 24 maj 03.27 getafix-hda.bimg-20150523-032706
-rw-------. 1 root root 3972202496 25 maj 03.27 getafix-hda.bimg-20150524-032708
-rw-------. 1 root root 5911937024 25 maj 11.34 getafix-hda.bimg-20150525-032702
-rw-r--r--. 1 root root 427588321280 23 maj 03.27 getafix-hda.qcow2
[root@dualix images]# qemu-img info getafix-hda.qcow2
image: getafix-hda.qcow2
file format: qcow2
virtual size: 767G (823023108096 bytes)
disk size: 398G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hda.bimg-20150523-032706
image: getafix-hda.bimg-20150523-032706
file format: qcow2
virtual size: 767G (823023108096 bytes)
disk size: 4.0G
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hda.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hda.bimg-20150524-032708
image: getafix-hda.bimg-20150524-032708
file format: qcow2
virtual size: 767G (823023108096 bytes)
disk size: 3.7G
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hda.bimg-20150523-032706
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hda.bimg-20150525-032702
image: getafix-hda.bimg-20150525-032702
file format: qcow2
virtual size: 767G (823023108096 bytes)
disk size: 5.5G
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hda.bimg-20150524-032708
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false

[root@dualix images]# ll getafix-hdb*
-rw-------. 1 root root 198144 23 maj 03.27 getafix-hdb.bimg-20150523-032706
-rw-------. 1 root root 458752 25 maj 03.27 getafix-hdb.bimg-20150524-032708
-rw-------. 1 root root 198144 25 maj 03.27 getafix-hdb.bimg-20150525-032702
-rw-r--r--. 1 root root 11324817408 22 maj 19.46 getafix-hdb.qcow2
[root@dualix images]# qemu-img info getafix-hdb.qcow2
image: getafix-hdb.qcow2
file format: qcow2
virtual size: 69G (74100654080 bytes)
disk size: 11G
cluster_size: 65536
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hdb.bimg-20150523-032706
image: getafix-hdb.bimg-20150523-032706
file format: qcow2
virtual size: 69G (74100654080 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hdb.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hdb.bimg-20150524-032708
image: getafix-hdb.bimg-20150524-032708
file format: qcow2
virtual size: 69G (74100654080 bytes)
disk size: 388K
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hdb.bimg-20150523-032706
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false
[root@dualix images]# qemu-img info getafix-hdb.bimg-20150525-032702
image: getafix-hdb.bimg-20150525-032702
file format: qcow2
virtual size: 69G (74100654080 bytes)
disk size: 196K
cluster_size: 65536
backing file: /var/lib/libvirt/images/getafix-hdb.bimg-20150524-032708
backing file format: qcow2
Format specific information:
compat: 1.1
lazy refcounts: false

Thanks for any suggestions!

The documentation is unclear on how to restore backups. It would be useful to document this clearly from an end-user perspective (preferably with an example command-line in the README).

Hi,
I've a physical drive attached to my VM. The script fails with "'error: unsupported configuration: source for disk 'vde' is not a regular file; refusing to generate external snapshot name'". Could attached physical drives be excluded from the backup?

BR
Michael

See #24
In a next release we could think about introducing an additional flag for offline images backup. Having the backup performed always, regardless of the mode specified by the user, could be a bit confusing.
Alternatively we could just exit with an error if -c is used.

Function get_snapshot_chain is called after consolidation, so it returns just current image of VM.
The following fix works for me:

diff --git a/fi-backup.sh b/fi-backup.sh
index a093df2..26474a1 100755
--- a/fi-backup.sh
+++ b/fi-backup.sh
@@ -426,6 +426,15 @@ function consolidate_domain() {
       if [ -n "$backing_file" ]; then
          print_v d "Parent block device: '$backing_file'"

+         snapshot_chain=()
+         #get an array of the snapshot chain starting from last child and iterating backwards
+         # e.g.    [0]     [1]      [2]     [3]
+         #       snap3 <- snap2 <- snap1 <- orig
+         #
+         # blockcommit: orig -> snap1 -> snap2 -> snap3 [becomes] orig
+         # blockpull:   orig -> snap1 -> snap2 -> snap3 [becomes] snap3
+         get_snapshot_chain "$block_device" snapshot_chain
+
          # Consolidate the block device
          #echo "ABOUT TO RUN:" 
          #echo "$VIRSH -q $CONSOLIDATION_METHOD $domain_name $block_device ${CONSOLIDATION_FLAGS[*]}"
@@ -438,15 +447,6 @@ function consolidate_domain() {
             return 1
          fi

-         snapshot_chain=()
-         #get an array of the snapshot chain starting from last child and iterating backwards
-         # e.g.    [0]     [1]      [2]     [3]
-         #       snap3 <- snap2 <- snap1 <- orig
-         #
-         # blockcommit: orig -> snap1 -> snap2 -> snap3 [becomes] orig
-         # blockpull:   orig -> snap1 -> snap2 -> snap3 [becomes] snap3
-         get_snapshot_chain "$block_device" snapshot_chain
-
          if [ "$CONSOLIDATION_METHOD" == "blockcommit" ]; then
             # --delete option for blockcommit doesn't work (tested on
             # LibVirt 1.2.16, QEMU 2.3.0), so we need to manually delete old

Thank you for your interesting and inspiring script. I was wondering, if it is possible to come around of the commands against the backup folder.

As far as I understand, the libvirt command for snapshots are used to create the multiple set of disk files on an incremental basis. The script uses "cp" for moving the different parts to the backup directory. The incremental qcow2 files are located next to the basis image.

For consolidating the files, virsh uses the the files around the basis image, but not against the backup directory.

Line 331ff seems mostly for creating some "paranoia checks" and then copy over the new file.

So, I'm asking & suggesting to make the "copy to backup folder" part optional or "different". And I would like to see rsync with --delete-after in place, working wildcard-based.. or something else like lftp.

My problem is, that I'm running in an environment on a dedicated root server and my backup space can only be addressed using FTP. For some reason FUSE-FTP delivers I/O errors.
I already setup "reoback" for doing incremental daily FTP backups, but that script fails with secondly changing images because of running domains and , ontop, copies the complete image file again & again, which wastes endless space.

What do you think?

best regards
Matthias

[ERR] Unsupported libVirt version '1.2.2'. Please use libVirt 0.9.13 or greather
[ERR] Unsupported /usr/bin/qemu-img version '2.0.0'. Please use 'qemu-img' 1.2.0 or greather
[ERR] Unsupported KVM version '2.0.0'. Please use KVM 1.2.0 or greather

I've been trying to get the latest builds to work with TravisCI and also with Docker in a scratch branch. Docker attempts failed too.

I can run tests locally and they run fine, but there seems to be a block at the TravisCI and hub.Docker.com blocking containerized libvirtd which is needed to automate the backup tests.

Suggestions?

when the VM has multiple disks, fi-backup fails with a "permission denied" error:

[DEB] Snapshotting block devices for 'ubuntuserver_clone' using suffix 'bimg-20190803-111757'
Formatting '/var/lib/libvirt/images/ubuntuserver_clone.bimg-20190803-111757', fmt=qcow2 size=17179869184 backing_file=/var/lib/libvirt/images/ubuntuserver_clone.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
[ERR] Snapshot for 'ubuntuserver_clone' failed! Exit code: 0
'error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied'

the second disk was added with virsh command:

virsh attach-disk ubuntuserver_clone /var/lib/libvirt/images/ubuntuserverclone-disk1-5G.qcow2 vdb --cache none

this happens regardless of the version of qemu, 2.5 or 2.11.

Thanks in advance for you help.

First: awesome script ... thanks for writing this!

For both backups and consolidation the script breaks the list of block_devices returned from "virsh domblklist" into block_device using:
for block_device in "$block_devices"; do
"for" will treat "$block_devices" (in quotes) as a single argument and assign all of the block devices to $block_device. I think what you meant was (no quotes):
for block_device in $block_devices; do

For example (with quotes):
# for i in "a b c"; do printf "arg: $i\n"; done
Returns:
"arg: a b c"

Where (without quotes):
# for i in a b c; do printf "arg: $i\n"; done
Returns:
arg: a
arg: b
arg: c