translate_el.sh: fail to chroot about compute-image-tools

[zmarano]

Is this an LVM disk or a disk with the boot loader on a separate partition from the root file system? If so, that would be why. Right now everything just fails miserably in those cases. We actually need to do a bunch more work here to sort out if we have mounted a valid root partition and where the boot loader is- if its not there maybe its on another partition etc. If you have any ideas how to sanely do that we're open :) I haven't come up with anything that makes sense yet.

[helen-fornazier]

I see, yes, I have /boot in a different partition indeed.
The rootfs in a linux system needs to have at least the file /sbin/init (imposed by the kernel), so it shouldn't be too difficult to identify which partition rootfs is by looping through all the partitions, mounting them and searching for this file, it should also be possible to mount a LVM disk and check its content (I'll do a better research on this).
For the boot partition, the script already assumes that grub is being used, so we could do the same and look for the grub.cfg file.
Another solution is to ask the user to run a helper/hint tool in the machine to analyze the system and upload the collected information to a gs bucket, but this requires one more interaction from the user (not good for UX).
We could also parse /etc/fstab, but if it uses labels instead of UUIDs for the disks, then it is not reliable, but we could use it as a hint.

[zmarano]

I was hoping to come up with some common library functions to do this- because all of the Linux imports need to figure out the same thing. I was looking at https://pypi.python.org/pypi/lvm2py/ as a possibility for LVM but it wasn't very helpful.

[helen-fornazier]

The only usage I see about using a common library is to retrieve the FS uuid or label, the partitions would still need to be mounted in the right location to allow chroot. Let me know if you were thinking in something else.
I was thinking in something like this https://gist.github.com/helen-fornazier/4cc431d79c523d8cfb4131e8d19c833f
i.e, to probe all the partitions, and then mount the rootfs and the boot in the right place to allow chroot.
This still need to be polished and discussed, but this would solve the LVM problem and also the /boot in a separated partition, the limitation is still if the system have other weird partitions, but this covers most of the cases.
What do you think?

[helen-fornazier]

We could also use guestfs to manage this.
I was able to mount the entire disk using
$ guestmount --pid-file guestmount.pid -a /dev/sdb -i /mnt
There are libraries in several languages, python and go included.
This tool uses the help of qemu to access the filesystem, it should be more reliable then the previous suggested approach.
There are some other ways to access the filesystem and to customize it without mounting it in the host using this tool, I am digging a bit more into this.

http://libguestfs.org/

[zmarano]

That looks promising. I never heard of guestfs- lets see if that will work for the common cases.

[zmarano]

I played around with the CLI tools a bit. virt-filesystems is sort of useful in that it can identify all the LVM volumes. It looks like you still need to mount everything and figure out where the boot loader lives. It is also a bit wonky in terms of displaying block devices with filesystems. For example, from a CentOS disk I have around.

~# virt-filesystems --filesystems -a /dev/sdb
/dev/sda1
/dev/centos/root

/dev/sda1 contains the boot loader (although in this case, its actually /dev/sdb1)
/dev/centos/root contains the root filesystem on an LVM volume

[helen-fornazier]

I tested with
$ guestmount -a /dev/sdb -i /mnt
I was able to complete the translate script with success, but I encountered other problems along the way:

• For Debian with LVM: It mounted the disk, ran the chroot commands and finished successfully, but the disk didn't boot, I think this is another problem not directly related to guestmount, but to the boot configs, I am still investigating this.

• For Centos 7: The script has other problems (wrong prefix and paths), and the boot config.
  I was able to finish the translate script and boot the machine partially because the major problem is when it has an initramfs which needs to be regenerated with the tool dracut, the problem is that dracut checks which kernel it runs so it can look for the modules at /lib/modules/<kern-version>, but as it is running through chroot, it get the wrong kernel information (basically it executes uname -r). I could probe which kernel version it uses and execute the command:
  $ dracut -f --kern-ver 3.10.327.el # For example
  But this doesn't seem ideal
  So I am checking the virt-customize tool, it uses qemu to access the system, and it allows me to execute a script inside the vm machine to install packages and do other changes. I am still investigating if it solves the entire problem but seems promising.
  So the idea would be, instead of mounting the disk and performing changes with chroot, we could use virt-customize directly to make all the changes, as it is in a isolated qemu environment, it should be more reliable.
  For example, it is possible to execute a command with:
  $ virt-customize -v -a /dev/sdb --run-command "ls -1 /"
  It shouldn't matter if the disk has multiple partitions with or without LVM. I am doing more tests.
  http://libguestfs.org/virt-customize.1.html#log-file

In the python-guestfs library, there is also the possibility to execute shell script inside the VM to customize it. So I was thinking we could write a script in bash, python or golang or any language you prefer that has a guestfs library (which language do you prefer?) and use virt-customize to mount and customize the image, if I understand correct (I am still doing more tests), this should deal with the LVM problems, several partition problems and also the uname -r problem.
What do you think?

[zmarano]

Oh yeah that worked!

Debian translations- are you sure its not booting or just not outputting anything to the serial console (which is a grub kernel cmd line option).

For CentOS, yes I found several bugs in the current script (will make a PR shortly to address those). This is the first time I have been able to test it on an actual imported disk. Will have to look into this dracut problem a bit more though.

I will have to think about the idea of using qemu- we don't currently have the ability of running a KVM VM inside of a GCE VM.

Language wise- I don't have a strong preference but given that we try to stay close to what our users would know, bash or python are preferable here. I did have intention to make common python libraries for all of these things and share them between the various translations.

[zmarano]

FYI: #143

[helen-fornazier]

The problem of using guestmount is that grub2-mkconfig generates the wrong rootfs parameter, so the system doesn't boot and only the dracut shell is loaded. Using virt-customize for this solves the problem, but it is not that easy to pass a bash script to virt-customize.
So I wrote a script in python that seems to solve most of our problems and seems to be working fine.
@zmarano please see #148

[zmarano]

Resolved by #148