zipurman / ovirt_simple_backup Goto Github PK

Expected behaviour

When I migrate from xenserver to ovirt, I attach the migrate disk to new VM and I expected it would work.

Actual behaviour

After migration, when I attach this disk to the new VM, there's no data (even no partitions)

Steps to reproduct the behaviour

oVirt version: 4.2.4.5-1.el7
XenServer version: 7.1

Steps to reproduce this issue:

• (oVirt) "Simple Backup" > "Xen Migrate"
• (oVirt) Choose the vm to migrate
• (oVirt) Type a new name for this VM
• (oVirt) Click the button "Migrate This Xen VM Now"

When it finishes, I attach the disk to the migrated machine.
I've tried to boot from an iso (CD, ...) and there is no partitions at all.

When backup :
the snapshot is created, but when trying to attach disk (I guess) there is a problem : No disk attached.

When restoring.

No data.xml or Disk Images found.

but thereis image.img and also data.xml

Migrate:

From xenserver side it works, it does all the steps and migrates the disk.
but when says that it is importing, it creates de VM but does not import the disk, so I have an empty VM.

Zipurman, hello!

I have only Fibre Channel storage. Can I use Fibre Channel LUN in SimpleBackup VM for /mnt/backups?
Should I create a NFS server on physical server? Maybe I can create a NFS server on SimpleBackup VM?
Please advise me a good practice for FC storage for backups.

After login to Ovirt webUI, this message appeared:
https://ibb.co/n0nkse
when i click ok. then it will go to github page, namely:
https://test-engine.domain.int/ovirt-engine/webadmin/plugin/simpleBackup/start.html
https://ibb.co/cTi8Ce
how to fix it?

Expected behavior

When you run backup on VM hard drive (snapshote) should be attached to VM and backup should start

Actual behaviour

When you run backup on VM hard drive (snapshote) is not attached to VM, Error:
VDSM vmname.domain command HotPlugDiskVDS failed: General Exception: ("Unsupported {u'write_bytes_sec': 5242880, u'total_iops_sec': 0, u'read_iops_sec': 100, u'read_bytes_sec': 62914560, u'write_iops_sec': 10, u'total_bytes_sec': 0} for ioTune",)

Steps to reproduce the behavior

Create a VM
Create a QOS and storage profile:
->Compute -> Data Centers -> Click on datacenter -> QOS tab -> create new storage QOS with limitation of Read Write and IOPs:
Storage -> Storage Domains -> click on storage domain name -> Disk Profiles tab -> associate QOS with profile
Attach storage profile to VM drive:
Compute -> Virtual Machines -> VM name -> Disks Tab -> click "edit" disk and change Disk profile
Start Backup

Solution

Remove QOS profile from snapshot or if possible try to mount it in different way

Expected behaviour

Output something like: Backup data partition it's full.

Actual behaviour

Output it's: "Error Creating Backup Directories - Check Permissions to ..."

Steps to reproduct the behaviour

Just try to backup something with the backup mount-point full.

I tried to restore a machine, but the only thing that is restored is the VM, but not the disk.

as far as I can see, the appliance attaches the disk of the VM I want to restore, its OK.
but it seems that the attached disk is not send to the Datastore, and remain attached to the appliance, so the VM is restored (only the specifications) but no disk attached.

I have investigated the vmrestoresingle.sh file, I noticed that there are value named $newdiskuuid and $newdiskuuid thant does not have value. So when the instruciont obuattachadisk $newvmuuid $newdiskuuid is executed nothing happends because the variables are empty.

looks like in line 151 in vmrestoresingle.sh is not receiving the data . checkvm="${obuapicallresult}"

maybe I'm wrong but that's what I have found.

I use centos7.0 as my engine backup server, when I backup a vm, first attach this vm snapshot, then can find a /dev/vdb in backup server, when dettach this vm snapshot, but /dev/vdb is not deleted. this question should how to solute?

on the webgui theres no "save" button, although I've seen that's there is the code related to.
by the way I tried to configure via CLI, I can set the parameters, but nothing works.

there's a problem in the documentation.

when wget https://raw.githubusercontent.com/zipurman/oVIRT_Simple_Backup/master/xen_migrate/xen_migrate.sh

I've searched it, and its in https://github.com/zipurman/oVIRT_Simple_Backup/blob/master/BashVersion/OtherTools/xen_migrate/xen_migrate.sh

also
chmod +e xen_migrate.sh
+e is not a valid parameter shouldn’t it be +x ?

it does not find the file.

Expected behaviour

DD/PV copy should use the right drive letter.

Actual behaviour

If the "simple-backup" VM have more then one disk (sda + sdb), backup process always copy data from sdb, it writes this to the log:

Disk Dat Write Disk1 - e94e4899-5c88-4a7d-979f-7f98b91f3e82 - true - virtio_scsi - 32212254720 - sdb

sda /
sdb /var
sdc (disk attached from the VM we are doing the backup)

NOTE: this is kind of problem if you intend to use /dev/sdb as attached storage for backup and not NFS mounted.

Steps to reproduct the behaviour

Just create SimpleBackup VM with two disks.

Hi, I'm surely missing something but I can't see the Xen VM list, I specified the Xen host IP and Xen migration VM IP and UUID. I also copied the ssh key and the xe vm-list remote command works perfectly from the command line, and it shows like 15 or 16 VMs.

Can you please enlighten me a bit about what can be missing in my settings or in my Xen host?

when exporting I have to set a new name to the exported machine, I set xt-php-XS, but I can see that the name in /mnt/migrate is "xen1.img"

also I cannot select a Cluster where to restore (but I think this is retated to another open issue"

Thanks

Expected behaviour

should be good to select disk to backup when saving and restoring

Actual behaviour

we have to backup/restore the full disks stack

Steps to reproduct the behaviour

need a checkbox to do so
thank you

I have created VM in thin provisioning on a FC storage, a small one, a Debian with 10GB of disk, once created it uses 3GB more or less.

I have made a backup of this VM, everything OK.

When restoring to the same FC storage, I cannot do it in provisioning state, If I try it does the restore, but I can see the machine but the disk is not there, so it does not start.

If I restore the VM with provisioning it works.

NEW VERSION WILL BE READY VERY SOON - BE PATIENT AS IT'LL BE WORTH WAITING FOR!

Reboot your BackupEngineVM - Disks Out Of Order.
I`m rebooting VM, but nothing change.

If on backupEngineVm installed oVirt Guest Agent - not saving settings. After push Save button displaying message Reboot your BackupEngineVM - Disks Out Of Order. If BackupEngineVm without oVirt Guest Aget - working fine.

Once backup is working well ;)

I tried a restore and its not working as expected.

1- select VM to restore. Adjust parameters and OK
2- Disk is done 100% a pop up appears "the vm should be restored . you will have to manually confirm the memory... " and OK, but nothing else appears to confirm anything... isn't it the first step ?
3- I can see that the system is doing stuff with a disk, creating and attaching

4- and finally fail what was it doing.

once it has failed I have investigated and tried the following stesps
a- I searched in Storage->Disks and I found the disk it was restored

b- I attached to a new VM
c-started it, but the VM does not boot :(

so by now, I can do backups, but I cannot restore them, I'm at 50% ;)

Expected behaviour

Xen Detach VM HD from source and attach to VMMigrate server in the XEN environment. The convert VM HD for import in to Ovirt.

Actual behaviour

Script does not detach the HD from the source VM. It then turns the source VM back on. Source HD is not attaching to VMMigrate VM. It then attempts to image the HD resulting is the following messages 6 times per second.
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached
[2018-05-28:16:12:13] Xen - Imaging - 0/1 - 0% Disk Not Attached

Steps to reproduct the behaviour

Create Server in XEN as migration host with attached NFS storage.
Create Backup Server in Ovirt and mount both migrate and backup NFS Storage.
Supply UUID for Backup Server from Ovirt and supply UUID of VMMigrate server from XEN in setup.
Run migration from VM listed in Simple_Backup.
Source VM is stopped, drive does not detach. Server starts back up and logs show as Disk Not Attached.

When you have more than 1 drive connected to VM:

$numberofimages = count( $disktypeget['avaliabledisks'] );
if ( $numberofimages > 1 && empty( $recovery ) ) {
        sb_pagedescription( 'The backup VM has too many disks attached. Please remove all but the OS disk in order to preform a backup.' );

Prevent you to do a backup

Solution:
```$numberofimages = $numberofimages-$number_of_drives````
You could add the option in settings that will take no only last drive letter but as well how many drive there is in system

Problem:
When you use multipathing in your operating system can be a lot of physical drives there is not enough options in setting for this.

Restrictions:
Every drive that is connected to operating system needs to have this same prefix for example sd[X]

after disk snapshot that contains LVM is removed from backup appliance we get I/O error as system still see LVM drives and can't access to it.

how to avoid it:
After snapshot is detached run:
dmsetup remove_all

To avoid problems with LVM could be nice if you can mention in install manual that groups and LVM names inside the appliance need to be unique, When you create all machines from templates and you have this same name for each lvm VM hangs when you detach snapshot .

the instruction

chown www-data:disk /dev/vdb

return "file o directory does not exist"

the instruction
chmod 777 /dev/sr0
return "file o directory does not exist"

altough the file /webUI/server/site/index.php
(that was reported in another issue)
lacks this code at the beginning, if not the web is seen like the '90s :P

<head>
<link rel="stylesheet" type="text/css" href="css/style.css">
</head>

At the beginning want to say that I use Centos so maybe my problems are related to some package in this operating system.

Problem

Log file on doesn't parse new line character, each log is written in this same line with "\n" at the end

Solution:
edit sb_log function in functions.php, change echo -n to echo -e

` function sb_log( $logtext ) {

    GLOBAL $settings;
    if ( ! empty( $settings['backup_log'] ) ) {
        $logtext = str_replace( '"', '', $logtext );

        $logtime = strftime( "[%Y-%m-%d:%H:%M:%S]" );

        $logdata = $logtime . ' ' . $logtext;

        //get last line of log
        exec( 'tail ' . $settings['backup_log'] . ' -n 1', $lastlogitem );
        $lastlogitem = $lastlogitem[0];
        $lastlogitem = preg_replace( '/\[[0-9\-:]+\] /i', '', $lastlogitem );

        //dont log if nothing has changed
        if ( ! strpos( " {$lastlogitem}", "{$logtext}" ) !== false ) {
            sleep( 0.5 );
            exec( 'echo -e "\n' . $logdata . '" >> ' . $settings['backup_log'] );
        } else {
            exec( 'echo -e "." >> ' . $settings['backup_log'] );
        }
        sleep( 0.5 );

    }
}`

After importing some VMs from KVM (via libvirt) the UUID would not match in backup_create_path.php.
example UUID 2712f337-54d7-a781-24cb-53c642e35bff

The solution was to edit reg.php

•  $UUIDv4  = '/^[0-9A-F]{8}-[0-9A-F]{4}-4[0-9A-F]{3}-[89AB][0-9A-F]{3}-[0-9A-F]{12}$/i';
  

•  $UUIDv4 = '/^[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}$/i';
  

Other people might experience this issue.

Hi, just writing here some instructions that should be included in the documentation.

when installing dependences in
3 On oVirt Server
https://github.com/zipurman/oVIRT_Simple_Backup#steps-to-setting-up-xen-migration-and-backup-scripts

nfs-common package is missing, if not you cannot mount NFS volumens, also you receive an "wrong FS file" error and you can spend all the day looking for that.

4 . on oVirtEngine VM
once installed the plugin https://github.com/zipurman/oVIRT_Simple_Backup/tree/master/plugin

you should do once installed an in /usr/share/ovirt-engine/ui-plugins/simpleBackup

chmod 777 -R *

if not, the button "Simple Backup" does not appear in the "ovirt-engine" WebUI

once the migration is done, this VM is stopped, deattached copied disk, and attached to the original VM,
it's possible to start again the VM migrate machine ? if not it remains stopped.

thanks

Expected behaviour

When going to "Simple Backup->Restore" directory "lost+found" should not be show.

Actual behaviour

When going to "Simple Backup->Restore" directory "lost+found" from filesystem appears.

Steps to reproduct the behaviour

Use the backup data filesystem root as backup mount-point.

Hello

First I would like to congratulate the project.

I have a doubt.

why does not it appear to back up the Hosted VM of oVirt Simple Backup VM?

Regards,

Hello.

Your project looks very interesting.
From https://github.com/zipurman/oVIRT_Simple_Backup/blob/master/webUI/README.md

VMs with Multiple Disks - Backup/Restore Not Yet Supported

When can I expect this functionality? There is a road map?

I want to integrating oVIRT-Simple-Backup in my hosted-engine ISO installer, only to provide a iso to customers, I want to follow hosted-engine setup but this is complex, have you some good advise?

Expected behaviour

auto backup php file will ignore cdrom devices

Actual behaviour

{"status":1,"reason":"Imaging Disk /mnt/backups/FILESRV01/db499873-c33a-4035-b8f6-7670767c8b7d/ovirt_20180404_085751/progress.dat","progress":0,"numberofdisks":2,"thisdisk":""}fdisk: cannot open /dev/sr0: Permission denied

Steps to reproduct the behaviour

su apache
php /var/www/html/automatedbackup.php
script hangs trying to access /dev/sr0

Please add when possible support for pbzip2, it's parallel bzip2 implementation, much faster.

Hi, I followed your instructions here
#6 (comment)

but I still cannot see the Domain and Clusters,
at this moment I have
"Storage Domains"

• ovirt-image-repository (image created in the installations process automatically)
• SD-_SYN-NFS (Data)
• SD-SYN-ISO (ISO)

Also I think is related the issue that I cannot see in "Single Backup" option I cannot see any machine.

Also, In Restore option I can see the machines, but when selected one to restore I have this error.

There's no license associated with the code - can you please add an open source license to it?

Expected behaviour

Backup should continue and log writing should continue even if we change the focus on the browser.

Actual behaviour

log output stops (var/log/simplebackup.log) and backup progress also

Steps to reproduct the behaviour

1. start the backup (manually)
2. change the focus on your browser to any other place in ovirt interface
3. log output stops (var/log/simplebackup.log) and backup progress also
4. if we go back to "view progress" in simplebackup, the backup "continues" and log output goes back

hi !
when executing the script is asked and NFS volume for the backup.

is needed that the package nfs-common has to be installed before the script, if not an error is received.

cheers !

in the single backup options, de "disk" size column shows wrong information

Hi, really a nice Backup Solution! Installer worked like a charm (except the Plugin Installation in oVirt which I will look after later).
When I restore the Disks are restored, but the VM not. Result is a new Disk without VM attached, I can't even see a request for creating am VM in the engine.log.
I'm testing with 4.2.3, here's the Log, any Idea what I did wrong?

2018-05-07:22:50:35] Imaging Disk(s) Completed
[2018-05-07:22:50:36] Restore Imaging: 1/1 - 0% Imaging Disk(s) Completed
[2018-05-07:22:50:38] Create New VM -
test1

Default


Blank

1073741824
<memory_policy>
4294967296
</memory_policy>


1
1
0





hd


windows_7

workstation

[2018-05-07:22:50:40] Update Graphic Console - spice
[2018-05-07:22:50:40] VM Create - VM Created
[2018-05-07:22:50:40] Detatching Disk From BackupEngine - c85f8fe6-f5f5-4a0f-81c9-07c30f0d27b2
[2018-05-07:22:50:45] BOOTABLE - 1
[2018-05-07:22:50:45] -- DISK ATTACH -- 1 bootable: true
[2018-05-07:22:50:45] <disk_attachment>


true
true
virtio_scsi
</disk_attachment>
[2018-05-07:22:50:46] VM Restore Completed

I followed the instructions for configuring the exportation from Xen.
I know that this worked for me at the first releases, but now, I think I missed something.

when I want to export a VM, I can see the process that the VMbackup rebots, and attach the desired disk, but when it tries to export it keep writing on the log "0% Imaging in progress."

[2018-04-11:13:28:21] Xen - Imaging - 1/1 - 0% Imaging in progress.
[2018-04-11:13:28:18] Xen - Imaging - 1/1 - 0% Imaging in progress.
[2018-04-11:13:28:15] Xen - Imaging - 1/1 - 0% Start Imaging
[2018-04-11:13:28:14] Xen - Imaging - /dev/xvday
[2018-04-11:13:28:13] Xen - Imaging - 0/1 - 0% No Expected Disk Available
[2018-04-11:13:28:10] Xen - Imaging - 0/1 - 0% No Expected Disk Available
[2018-04-11:13:27:59] Xen - Start - Started
[2018-04-11:13:27:57] Xen - Start - Starting
[2018-04-11:13:27:46] Xen - Add VBD - Disks Attached
[2018-04-11:13:27:43] Xen - Add VBD - Adding
[2018-04-11:13:27:40] Xen - Shutdown - Halted
[2018-04-11:13:27:39] Xen - Shutdown - Shutting Down
[2018-04-11:13:27:33] Xen - Remove VBD - Removed
[2018-04-11:13:27:31] Xen - Shutdown - Halted
[2018-04-11:13:27:28] Starting Xen Migrate ... 71a6d3d1-e8a2-f4e0-01bf-6bbe7549c0dd
[2018-04-11:13:27:28] ------------------------------------------------------ 

As you told me, I think it would be very helpfull to write down this advertising on the documentation, its another pretty way to lose days with this issue =P

BackupEngine MUST not have snapshots

hi.
when trying

vi /etc/apache2/sites-available/default-ssl.conf

    ServerName backupengine.yourdomain.com:443
    DocumentRoot /var/www/html/site
    SSLCertificateFile /etc/apache2/ssl/apache.crt

and after doing all the following steps, I have 2 problems.
if I put the certificate as related, the apache does not start.
otherwise if I don't change the certificate, then navigator (firefox) does not allow me to open than web because is a "self signed certificate"

using debian 9 . php7

Expected behaviour

It should scsi remove in Kernel the attached disks before asking engine to remove them.

I think we need to do some exec(echo 1 > /sys/block/sdb/device/delete), I checked comm/disk_detatch.php code, should be there but I did not find the driver letter variable (check NOTES section).

Actual behaviour

They are removed directly in engine without behing removed on Linux Kernel, this results in stuff like this:
[ 3819.288384] Buffer I/O error on dev dm-1, logical block 0, async page read
[ 3819.290094] Buffer I/O error on dev dm-2, logical block 0, async page read
[ 3819.291579] Buffer I/O error on dev dm-3, logical block 0, async page read
[ 3819.293523] Buffer I/O error on dev dm-4, logical block 0, async page read
[ 3819.293937] Buffer I/O error on dev dm-5, logical block 0, async page read
[ 3819.294655] blk_update_request: I/O error, dev sdf, sector 0
[ 3819.294896] Buffer I/O error on dev sdf, logical block 0, async page read

Notes

I think all the logic on the backup code settles over that the drives are "always" sequential, sdc, sdd, sde, ... if for some reason and somehow this does not happen (for numerous reasons), the backup logic all breaks, I got so many errors that even got Kernel Panic in SimpleBackup VM sometimes...

We need some way to identify directly which disk it's which outside and inside the SimpleBackup VM. This would help on future also:

• We would not need to know "which drive starts" in the settings, meaning I could have as many disks belonging SimpleBackup OS as wanted;
• We could make simultaneous backup;
• If for some reason the backup breaks and the kernel freaks on the drive letters ordering, the next backup would work anyway;
• per-vm disk exclude disks, we sometimes don't need to backup all disks in all VMs (eg: fileserver data disks are already backup with rsnapshot);
• Support storage LUN attached to SimpleBackup VM and mounted in /mnt/backup and not only NFS share;

Enhancement

It would be great to be able to do also snapshot only backups

I know this in not really a backup, but I would like to be able to:

• Schedule snapshot creation
• Ability to set a retention and automatically delete snapshot's

If it is too complex to include, I wouldn't mind to do recovery directly on oVirt.

[2019-02-26:08:47:31] Deleting Snapshot - Deleting Snapshot
[2019-02-26:08:47:31] Backup Completed - BU_20190226_084501
[2019-02-26:08:47:31] ---- Automated Backup Done
[2019-02-26:08:47:31] ----------------------
[2019-02-26:08:47:31] ----------------------
[2019-02-26:08:47:26] Detatching Image - Disk(s) Detatched
[2019-02-26:08:47:20] Backup Imaging - /0 - 100% Imaging Disk(s) Completed
[2019-02-26:08:47:07] Attaching Image - Disk(s) Attached
[2019-02-26:08:47:06] Disk Dat Write srv2_disk_01 - 2e512373-9a3e-4822-af54-f3b377cc0522 - true - virtio_scsi - 10737418240 - sdz
[2019-02-26:08:47:00] Attach Disk 2e512373-9a3e-4822-af54-f3b377cc0522
[2019-02-26:08:46:55] Creating Backup Paths - Directories Available
[2019-02-26:08:46:29] Creating Snapshot
[2019-02-26:08:46:27] Starting Backup ... 2aa0c2f6-6b17-4f82-8935-27d08b1ff5fd srv2.domain.name
[2019-02-26:08:46:27] ------------------------------------------------------
[2019-02-26:08:46:25] Deleting Snapshot - Deleting Snapshot
[2019-02-26:08:46:25] Backup disk space: 1%
[2019-02-26:08:46:25] Backup Completed - BU_20190226_084501
[2019-02-26:08:46:25] Backing up VM UUID: 2aa0c2f6-6b17-4f82-8935-27d08b1ff5fd
[2019-02-26:08:46:20] Detatching Image - Disk(s) Detatched
[2019-02-26:08:46:14] Backup Imaging - /0 - 100% Imaging Disk(s) Completed
[2019-02-26:08:46:00] Attaching Image - Disk(s) Attached
[2019-02-26:08:45:59] Disk Dat Write srv1_disk_01 - e5323472-4ee7-4af7-8271-05cef34ddeee - true - virtio_scsi - 42949672960 - sdz
[2019-02-26:08:45:54] Attach Disk e5323472-4ee7-4af7-8271-05cef34ddeee
[2019-02-26:08:45:49] Creating Backup Paths - Directories Available
[2019-02-26:08:45:06] Creating Snapshot
[2019-02-26:08:45:04] Starting Backup ... 8e1db77e-7f71-4e9e-afac-e98d573acfc2 srv1.domain.name
[2019-02-26:08:45:04] ------------------------------------------------------
[2019-02-26:08:45:02] Backup disk space: 1%
[2019-02-26:08:45:02] Backing up VM UUID: 8e1db77e-7f71-4e9e-afac-e98d573acfc2
[2019-02-26:08:45:02] Automated Backup - DailyBackup - Starting ....
[2019-02-26:08:45:02] ----------------------
[2019-02-26:08:45:02] ----------------------

I set an exception to firefox( Add Exception: FireFox -> Tools -> Advanced -> View Certificates -> Servers -> Add Exception.
), so I can avoid the "self-signed certificate" error.

but now I cannot see the web correctly.
Seems that the main page is not finding the header, style, footer...

in the WebUI setting there is "oVirtEngine user" twice. the second one should be "oVirtEngine password"

Expected behaviour

I want to do a single backup on datacenters that contains a lot of vm (between 100 and 150). List vms should display so as to select the vm I want to backup

Actual behaviour

blank page is returned

Steps to reproduct the behaviour

works as expected for smaller datacenters (about 40 vms) whereas it is long to display.

When trying to do a backup I receive this error

[2018-04-07:22:46:52] Backup Imaging - /0 - 0% Disk Not Attached
[2018-04-07:22:44:18] Backup Imaging - /0 - 0% Disk Not Attached
[2018-04-07:22:44:17] Attaching Image - Disk(s) Attached
[2018-04-07:22:44:08] Creating Backup Paths - Directories Available
[2018-04-07:22:43:55] Creating Snapshot
[2018-04-07:22:43:54] Starting Backup ... 83fbd444-b049-4624-ad5d-48f6baf6128e dummy-debian9
[2018-04-07:22:43:54] ------------------------------------------------------