kayrus / deploy-vm

Libvirt wrapper to spawn VMs using cloud images

Install libvirt on Ubuntu

sudo apt-get install -y libvirt-bin virtinst qemu-kvm virt-manager git wget genisoimage
sudo service libvirt-bin start

For the Windows VM support install bsdtar (this tool allows to extract zip archive from stdin):

sudo apt-get install bsdtar

Install on Fedora/CentOS

sudo yum install -y libvirt virt-install qemu-kvm virt-manager git wget genisoimage NetworkManager
sudo service libvirtd start

For the Windows VM support install bsdtar (this tool allows to extract zip archive from stdin):

sudo yum install bsdtar

This string inside your ~/.profile will allow you to use virsh:

export LIBVIRT_DEFAULT_URI=qemu:///system

Configure local resolver to use libvirt's dnsmasq

• Ubuntu 18.04 (systemd-resolved (c))

The setting below is not preserved on reboot:

systemd-resolve --set-dns=192.168.122.1 --set-domain=vm --interface=virbr0

Verify:

systemd-resolve --status virbr0

Add the following section into libvirt default network (sudo virsh net-edit default):

  <domain name='vm' localOnly='yes'/>

And restart the network:

sudo virsh net-destroy default
sudo virsh net-start default

• Ubuntu/Debian

virsh net-dumpxml default | sed -r ":a;N;\$!ba;s#.*address='([0-9.]+)'.*#nameserver \1#" | sudo tee -a /etc/resolvconf/resolv.conf.d/head && sudo resolvconf -u

• Fedora/CentOS

sudo systemctl enable NetworkManager
echo -e "[main]\ndns=dnsmasq" | sudo tee -a /etc/NetworkManager/NetworkManager.conf
virsh net-dumpxml default | sed -r ":a;N;\$!ba;s#.*address='([0-9.]+)'.*#server=\1\nall-servers#" | sudo tee /etc/NetworkManager/dnsmasq.d/libvirt_dnsmasq.conf
sudo systemctl restart NetworkManager

Add current user into libvirt group (will allow you to run scripts without sudo)

sudo usermod -aG libvirtd $USER # for Debian/Ubuntu14.04/Ubuntu16.04
sudo usermod -aG libvirt $USER # for CentOS/Fedora/Ubuntu18.04

NOTE: You have to relogin into your UI environment to apply these changes.

Allow libvirt to read VMs images in your home directory

ACL solution

Add permissions

setfacl -m "u:libvirt-qemu:--x" $HOME # for Debian/Ubuntu
setfacl -m "u:qemu:--x" $HOME # for CentOS/Fedora

Remove permissions

Remove ACL entries only for libvirt

setfacl -m "u:libvirt-qemu:---" $HOME # for Debian/Ubuntu
setfacl -m "u:qemu:---" $HOME # for CentOS/Fedora

Remove all custom ACL entries

setfacl -b $HOME
getfacl $HOME

Groups solution

Add permissions

sudo usermod -aG $USER libvirt-qemu # for Debian/Ubuntu
sudo usermod -aG $USER qemu # for CentOS/Fedora
chmod g+x $HOME

Remove permissions

sudo usermod -G "" libvirt-qemu # for Debian/Ubuntu
sudo usermod -G "kvm" qemu # for CentOS/Fedora
chmod g-x $HOME

Configure virsh environment

echo "export LIBVIRT_DEFAULT_URI=qemu:///system" >> ~/.bashrc

Configure ~/.ssh/config

cat dot_ssh_config >> ~/.ssh/config
chmod 600 ~/.ssh/config

Run Flatcar VMs cluster of 3 nodes

./deploy_flatcar_cluster.sh -s 3

user_data file works only for Flatcar and contains a template for Flatcar configuration and it configures etcd2 and fleet.

Try out Tectonic

Create Tectonoic credentials files:

• tectonic.lic # raw base64 encoded licence
• docker.cfg # raw base64 encoded dockercfg

Deploy cluster:

./deploy_k8s_cluster.sh --tectonic

Enter your Kubernetes master node:

ssh core@k8s-master # [-i ~/.ssh/id_rsa]

Get Tectonic [email protected] password:

kubectl --namespace=tectonic-system get secret tectonic-identity-admin-password -o template --template="{{.data.password}}" | base64 -d && echo

Login Tectonic:

https://k8s-master:32000

Username: [email protected] Password: see above

Run other VMs

Linux

Run three CentOS VMs

./deploy_vms_cluster.sh -o centos -s 3

Windows

Run one Windows IE11.Win7 VM

./deploy_vms_cluster.sh -o windows -r IE11.Win7 -m 1024 -u 2

FreeBSD guest (experimental)

Download and run FreeBSD:

./deploy_vms_cluster.sh -o freebsd # for 10.3 RELEASE
./deploy_vms_cluster.sh -o freebsd -c 11.0 -r RC1 # for 11.0 RC1

FreeBSD QEMU images don't support cloud-init, so you have to configure network and ssh manually through the console:

dhclient vtnet0
echo 'sshd_enable=YES' >> /etc/rc.conf
echo 'ifconfig_DEFAULT=DHCP' >> /etc/rc.conf
echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
service sshd start
passwd

FreeBSD host (experimental)

pkg install bash wget bzip2 gnupg cdrtools libvirt qemu virt-manager
kldload vmm
libvirtd -d
ifconfig tap create
ifconfig bridge create
ifconfig bridge0 addm tap0 up

virt-install version > 1.4 should use following parameters:

--nographic \
--console nmdm,source.master=/dev/nmdm0A,source.slave=/dev/nmdm0B \
--network bridge=bridge0 \

and without --vnc flag. VNC flag produces following XML:

<graphics type="vnc" port="-1" keymap="en-us"/>

which causes internal error: cannot determine default video type error message.

Completely destroy and remove all related VMs cluster data

./remove_cluster.sh flatcar

deploy-vm's Issues

Add parameters handler

Add wait time for the cluster up and ready

while true ping, etc.

Make scripts work with Ubuntu apparmor

Running scripts without sudo with the images directory inside user homedir causes this error message:

Starting install...
ERROR    internal error: process exited while connecting to monitor: qemu-system-x86_64: -drive file=/home/user/.libvirt/centos/centos1.qcow2,if=none,id=drive-virtio-disk0,format=qcow2: could not open disk image /home/user/.libvirt/centos/centos1.qcow2: Could not open backing file: Could not open '/home/user/.libvirt/centos/CentOS-7-x86_64-GenericCloud.img': Permission denied

Even when this directory has read permissions.

These commands resolve the issue:

$ sudo apt-get install apparmor-profiles apparmor-utils
$ sudo aa-complain /usr/lib/libvirt/virt-aa-helper

Update downloaded image when it is outdated

using curl -z out.img -o out.img URL. It is necessary to check whether current VMs use outdated image to avoid conflict.

Add gpg signature checking for the images

subj

use MAC address to identify VMs which don't provide hostname to dnsmasq

subj

Add rebuild ISO script inside VM_HOSTNAME dir

so it will be easy to rebuild ISO image from the openstack dir source

Add possibility to run scripts without sudo

using virsh pool, etc.

Manage --os-type and --os-variant for virt-install

Add suse support

Fix selinux checker

subj

Add prefixes for the VMs names/hostnames

subj

Add freebsd support

https://pellaeon.github.io/bsd-cloudinit/