dmacvicar / terraform-provider-libvirt

Terraform provider to provision infrastructure with Linux's KVM using libvirt

Terraform provider for libvirt

• (IRC gateway)
• Planning board: Github Projects

---

This is a terraform provider that lets you provision servers on a libvirt host via Terraform.

Introduction & Goals

This project exists:

• To allow teams to get the benefits Software Defined Infrastructure Terraform provides, on top of classical and cheap virtualization infrastructure provided by Linux and KVM This helps in very dynamic DevOps, Development and Testing activities.
• To allow for mixing KVM resources with other infrastructure Terraform is able to manage

What is NOT in scope:

• To support every advanced feature libvirt supports

  This would make the mapping from terraform complicated and not maintainable. See the How to contribute section to understand how to approach new features.

Getting started

The provider is available for auto-installation from the Terraform Registry.

In your main.tf file, specify the version you want to use:

terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
    }
  }
}

provider "libvirt" {
  # Configuration options
}

And now run terraform init:

$ terraform init

Creating your first virtual machine

Here is an example that will setup the following:

• A virtual server resource

(create this as main.tf and run terraform commands from this directory):

provider "libvirt" {
  uri = "qemu:///system"
}

You can also set the URI in the LIBVIRT_DEFAULT_URI environment variable.

Now, define a libvirt domain:

resource "libvirt_domain" "terraform_test" {
  name = "terraform_test"
}

Now you can see the plan, apply it, and then destroy the infrastructure:

$ terraform init
$ terraform plan
$ terraform apply
$ terraform destroy

Look at more advanced examples here and check the documentation.

Manual installation

You can also manually download the provider from the releases section on Github. To install it, refer to the Terraform documentation.

Building from source

• Go is required for building.

git clone https://github.com/dmacvicar/terraform-provider-libvirt.git
cd terraform-provider-libvirt
make

The binary will be called terraform-provider-libvirt.

Using multiple hypervisors / provider instances

You can target different libvirt hosts instantiating the provider multiple times. Example.

Using qemu-agent

From its documentation, qemu-agent:

It is a daemon program running inside the domain which is supposed to help management applications with executing functions which need assistance of the guest OS.

Until terraform-provider-libvirt 0.4.2, qemu-agent was used by default to get network configuration. However, if qemu-agent is not running, this creates a delay until connecting to it times-out.

In current versions, we default to not to attempt connecting to it, and attempting to retrieve network interface information from the agent needs to be enabled explicitly with qemu_agent = true, further details here. Note that you still need to make sure the agent is running in the OS, and that is unrelated to this option.

Note: when using bridge network configurations you need to enable the qemu_agent = true. otherwise you will not retrieve the ip addresses of domains.

Be aware that this variables may be subject to change again in future versions.

Upstream projects using terraform-libvirt:

• sumaform sumaform is a way to quickly configure, deploy, test Uyuni and SUSE Manager setups with clients and servers.

• ha-cluster-sap Automated HA and SAP Deployments in Public/Private Clouds (including Libvirt/KVM)

• ceph-open-terrarium ceph-open-terrarium is a way to quickly configure, deploy, tests CEPH cluster without or with Deepsea

• kubic

  • kubic-terraform-kvm Kubic Terraform script using KVM/libvirt

• Community Driven Docker Examples Docker examples showing how to use the Libvirt Provider

• Openshift 4 Installer The Openshift 4 Installer uses Terraform for cluster orchestration and relies on terraform-provider-libvirt for libvirt platform.

• Kubitect - a CLI tool for deploying and managing Kubernetes clusters on libvirt platform.

Authors

• Duncan Mac-Vicar P. [email protected]

See also the list of contributors who participated in this project.

The structure and boilerplate is inspired from the Softlayer and Google Terraform provider sources.

License

• Apache 2.0, See LICENSE file

terraform-provider-libvirt's Issues

Implement If-Modified-Since when downloading images

Our HTTP client currently downloads the image specified on source even when the image in the server is the same as a local copy previously downloaded.

For ths case where a local copy is already in the pool, we could perform the HTTP request with the If-Modified-Since header (using the modification time of the local image), so it would be downloaded only if the remote file has been updated.

Feature Request: Target Multiple Hypervisors

Currently the uri of the hypervisor is defined in the provider block. Since terraform only allows one provider per config it's not possible to target more than one hypervisor.

provider "libvirt" {
  uri = "qemu:///system"
}

This feature would make it possible to deploy multiple VM's across multiple hosts with a single apply operation. This is especially useful for clustered applications such as Kubernetes.

We should wait until the domain is up

Upon creation, we should wait that the domain is in state RUNNING and also may be wait that there are no pending jobs associated to it.

Network: adresses not assigned to VM

When network_interface is defined is supposed to be statically assigned an IP with addresses key and a list of IPs.

network_interface {
    network_name = "ose"
    hostname ="node${count.index}.osc.test"
    mac = "52:54:00:00:00:a${count.index + 1}"
    addresses = ["192.168.100.10${count.index + 1}"]

When terraform apply is launched the IP for the VM is retrieved via DHCP and is different from the one defined in addresses

Issues with Apparmour on Ubuntu 16.04. Permission denied

Hello, thanks for your work.

I'm trying to create VM with volume and network, but following getting error:

libvirt_domain.domain: Error creating libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: 2017-02-14T00:37:50.691878Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/rs-vol-0,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/var/lib/libvirt/images/rs-vol-0': Permission denied

Root cause was in Apparmor:

Feb 14 01:37:50 cz7825 audit[14624]: AVC apparmor="DENIED" operation="open" profile="libvirt-ac2850aa-7938-4e68-bb81-57cfeeba458b" name="/var/libvirt/images/rs-vol-0" pid=14624 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Feb 14 01:37:50 cz7825 audit[14624]: AVC apparmor="DENIED" operation="open" profile="libvirt-ac2850aa-7938-4e68-bb81-57cfeeba458b" name="/var/libvirt/images/rs-vol-0" pid=14624 comm="qemu-system-x86" requested_mask="wr" denied_mask="wr" fsuid=111 ouid=111

Libvirt for some reason didn't mention disk files in apparmor configuraion for this VM (Same happens with qemu-monitor device also). Other VMs (not managed by Terraform) work fine and have appropriate permissions in Apparmor. Checked on multiple hosts w/ 16.04.

# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
  "/var/log/libvirt/**/rs-0.log" w,
  "/var/lib/libvirt/qemu/domain-rs-0/monitor.sock" rw,
  "/var/run/libvirt/**/rs-0.pid" rwk,
  "/run/libvirt/**/rs-0.pid" rwk,
  "/var/run/libvirt/**/*.tunnelmigrate.dest.rs-0" rw,
  "/run/libvirt/**/*.tunnelmigrate.dest.rs-0" rw,
  # for qemu guest agent channel
  owner "/var/lib/libvirt/qemu/channel/target/domain-rs-0/**" rw,

Template is following:

provider "libvirt" {
  uri = "qemu+ssh://root@pdt1/system"
}

variable "num_nodes" {
  default = 1 
}

variable "prefix" {
  default = "rs"
}

resource "libvirt_network" "network" {
  name = "${var.prefix}-net"
  domain = "lab"
  mode = "none"
}

resource "libvirt_volume" "volume" {
  name = "${var.prefix}-vol-${count.index}"
  size = "${100 * 1024 * 1024 * 1024}"
  count = "${var.num_nodes}"
}

resource "libvirt_domain" "domain" {
  name = "${var.prefix}-${count.index}"
  vcpu = 1 
  memory = 3072
  disk {
    volume_id = "${element(libvirt_volume.volume.*.id, count.index)}"
  }
  network_interface {
    network_id = "${libvirt_network.network.id}"
  }
  count = "${var.num_nodes}"
}

Version:

• Ubuntu 16.04.1
• libvirt-bin 1.3.1-1ubuntu10.6
• apparmor 2.10.95-0ubuntu2.5
• Terraform v0.8.6

Second VM always fails to start, but starts if tried manually using `virsh`

If I am provisioning multiple VM's, the first one boots as expected but the second VM throws an error by default:

Error applying plan:

2 error(s) occurred:

* libvirt_domain.saltminion: Error creating libvirt domain: [Code-1] [Domain-10] internal error: qemu unexpectedly closed the monitor: ((null):25956): Spice-Warning **: reds.c:2484:reds_init_socket: reds_init_socket: binding socket to 127.0.0.1:5900 failed
2016-09-11T09:49:37.443038Z qemu-system-x86_64: failed to initialize spice server
* libvirt_domain.saltmaster.1: Error creating libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: ((null):25954): Spice-Warning **: reds.c:2484:reds_init_socket: reds_init_socket: binding socket to 127.0.0.1:5900 failed
2016-09-11T09:49:37.447608Z qemu-system-x86_64: failed to initialize spice server

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

I tried manually starting the VM's using virsh and it gracefully boots. I swapped the order and confirmed that it is the second VM. I tried multiple copies of the same VM and the same error again.

libvirt.tf:

provider "libvirt" {
    uri = "qemu:///system"
}

resource "libvirt_volume" "opensuse_leap" {

    name = "opensuse_leap"
    source = "http://download.opensuse.org/repositories/Cloud:/Images:/Leap_42.1/images/openSUSE-Leap-42.1-OpenStack.x86_64.qcow2"

}

resource "libvirt_volume" "mastervolume" {

    name = "saltmaster-${count.index}"
    base_volume_id = "${libvirt_volume.opensuse_leap.id}"
    count = 1
}

resource "libvirt_volume" "minionvolume" {

    name = "saltminion-${count.index}"
    base_volume_id = "${libvirt_volume.opensuse_leap.id}"
    count = 1
}

resource "libvirt_domain" "saltmaster" {

    name = "saltmaster-${count.index}"
    disk {
        volume_id = "${element(libvirt_volume.mastervolume.*.id, count.index)}"
    }
    count = 1
}

resource "libvirt_domain" "saltminion" {

    name = "saltminion-${count.index}"
    disk {
        volume_id = "${element(libvirt_volume.minionvolume.*.id, count.index)}"
    }
    count = 1
}

It could potentially be the issue where this plugin is trying to register the new VM with the same VNC endpoint localhost:5900 which IMHO is unlikely. AFAIK, libvirt should handle this part of the configuration of the given VM.

I am using Arch Linux updated to latest versions and there is no problem with other deployment frameworks that I am using to drive VM's in libvirt/KVM. I am quite sure that this issue is not arising from my choice of the distro and environment.

Limited options for cloud-init

It would be great to have cloud-init option to inject custom scripts in the cluster. Right now that is not supported and it gets difficult to predict the IP addresses for the VM's in case of a dynamic cluster.

cpu-model selection

I just realized that my terraform VM are now spawned with the emulator CPU type.
I tried to figure out how to make qemu/libvirt use 'host-model' as default to workaround that.
My VM are very slow at performing key computations right now.
Do you have any workaround for that or plan to support specifying it ?

error starting VM with a second disk

Whenever I start a VM with more than 1 disk attached I get:

• libvirt_domain.centos6_vm1_domain: Error crearing libvirt domain: [Code-27] [Domain-20] XML error: target 'sda' duplicated for disk sources '/home/vmstore/images/centos6_vm1disk_44_data' and '/home/vmstore/images/centos6_vm1disk_44'

using libvirt-daemon-kvm-1.3.0-1.el7.x86_64

module.vm_dbs_redundant.libvirt_domain.centos6_vm1_domain: Creating...
disk.#: "" => "2"
disk.0.volume_id: "" => "/home/vmstore/images/centos6_vm1disk_44"
disk.1.volume_id: "" => "/home/vmstore/images/centos6_vm1disk_44_data"
memory: "" => "6144"
name: "" => "centos6_vm_domain_44"
network_interface.#: "" => "1"
network_interface.0.mac: "" => "52:54:00:A1:A3:44"
network_interface.0.network: "" => "host-bridge2"
vcpu: "" => "2"

Is that working properly on your side ?

Failed apply doesn't clean up domains

On my machine terraform apply fails pretty often, and when terraform apply fails the domains that failed to set up are "stopped" but are not "undefined". So subsequent applys will fail because the domains already exist. Is there a way for us to do cleanup when apply fails?

ssh key resources

The main idea is to allow ssh key resources, and then locally assemble the cloud init metadata, wrap it into an iso and create a metadata volume for the instance resource uploading this created image.

cloud-init can use this disk as a metadata/userdata source to configure the ssh key.

Unsupported machine type

When running terraform apply with the example libvirt.tf I get the following error:

Error applying plan:

1 error(s) occurred:

* libvirt_domain.terraform_test: Error crearing libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: 2016-04-05T09:56:21.548490Z qemu-system-x86_64: -msg timestamp=on: Unsupported machine type
Use -machine help to list supported machines!

It seems to be caused because the defined machine pc-i440fx-2.4 in domain_def.go is not there for the following libvirt version:

$ virsh --version
1.2.18.2

Changing the value to an existing machine fixes the problem as expected.

false positive in network_interface change detection forcing rebuild of domain

Hi.

Since I pulled that latest master this morning, I no longer see storage pool issues.

However, even if I don't make any change to the network_interface data., the provider always sees:

network_interface.#:                "0" => "1" (forces new resource)
network_interface.0.address.#:      "" => "<computed>"
network_interface.0.mac:            "" => "52:54:00:A1:A3:61" (forces new resource)
network_interface.0.network:        "" => "host-bridge2" (forces new resource)
network_interface.0.wait_for_lease: "" => "0" (forces new resource)

As a result, my VM are being refreshed or completely rebuilt, this is not convenient.
I wonder if this is expected.
Thank you.

Can't compile from latest master branch

Attempted to recompile against the latest master and got the following errors:

github.com/dmacvicar/terraform-provider-libvirt/libvirt

libvirt/resource_libvirt_domain.go:212: domain.ListAllInterfaceAddresses undefined (type libvirt.VirDomain has no field or method ListAllInterfaceAddresses)
libvirt/resource_libvirt_domain.go:212: undefined: libvirt.VIR_DOMAIN_INTERFACE_ADDRESSES_SRC_LEASE
libvirt/resource_libvirt_domain.go:239: undefined: libvirt.VIR_IP_ADDR_TYPE_IPV4
libvirt/resource_libvirt_domain.go:241: undefined: libvirt.VIR_IP_ADDR_TYPE_IPV6
libvirt/resource_libvirt_domain.go:359: domain.ListAllInterfaceAddresses undefined (type libvirt.VirDomain has no field or method ListAllInterfaceAddresses)
libvirt/resource_libvirt_domain.go:359: undefined: libvirt.VIR_DOMAIN_INTERFACE_ADDRESSES_SRC_LEASE

[Domain-44] invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm

Hi.
I'm running my tests from a LVM VM and I don't have KVM nesting support.
It seems to me that I should use the QEMU emulation as reported by:
[terraform]$ virt-host-validate
QEMU: Checking for hardware virtualization : WARN (Only emulated CPUs are available, performance will be significantly limited)
QEMU: Checking for device /dev/vhost-net : WARN (Load the 'vhost_net' module to improve performance of virtio networking)
QEMU: Checking for device /dev/net/tun : PASS
LXC: Checking for Linux >= 2.6.26 : PASS

It looks like I need to change the hard coded defaults and recompile the module. I'm going to try that.
I might look into the code to find how to to make that a configuration parameter. But I'm a GO-lang ignorant right now..

Support for uploading images from local files

Only http remote resources are supported right now.

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/resource_libvirt_volume.go#L107
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/resource_libvirt_volume.go#L174

It should be trivial given that we use io.Copy, but we need to abstract the code path to get the size and then open the buffer that is copied.

Network in nat mode keeps recreating

This behavior only for nat mode. e.g. "none" mode works fine.

tf|⇒ terraform apply
libvirt_network.adm_net: Refreshing state... (ID: ec1b7cd1-4056-495f-bc57-71f87eb112be)
libvirt_network.adm_net: Destroying...
libvirt_network.adm_net: Destruction complete
libvirt_network.adm_net: Creating...
  addresses.#: "" => "1"
  addresses.0: "" => "10.30.0.1/24"
  bridge:      "" => "<computed>"
  domain:      "" => "lab"
  mode:        "" => "nat"
  name:        "" => "rs-terraform-test"
  running:     "" => "true"
libvirt_network.adm_net: Still creating... (4s elapsed)
libvirt_network.adm_net: Creation complete

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Terraform file

provider "libvirt" {
  uri = "qemu+ssh://root@pdt2/system"
}

resource "libvirt_network"
"adm_net" {
  name = "rs-terraform-test"
  domain = "lab"
  mode = "nat"
  addresses = ["10.30.0.1/24"]
}

Plugin built from aeaee86
Terraform v0.8.6
Libvirt version 1.3.1-1ubuntu10.7

Problem creating bridged network

resource "libvirt_network" "vm_network" {
  name = "vm_network"
  domain = "xx.xx.com"
  mode = "bridge"
  bridge = "br3"
}

gives the following libvirt error

* libvirt_network.vm_network: Error defining libvirt network: [Code-27] [Domain-19] XML error: The <nat> element can only be used when <forward> 'mode' is 'nat' in network vm_network -   <network>
      <name>vm_network</name>
      <domain name="london.zelotus.com"></domain>
      <bridge name="br3" stp="on"></bridge>
      <forward mode="bridge">
          <nat>
              <port start="1024" end="65535"></port>
          </nat>
      </forward>
  </network>

I've been looking at the source but sadly do not know go very well ( at all ) so haven't got very far.
I am assuming i need to pass an empty NAT block but can't work out the format ?
Documentation is not very forthcoming on NAT, so I'll volunteer to update or add some examples.

Thanks

Allow to define network interfaces for a domain

Mockup:

resource "libvirt_domain" "acceptance-test-domain" {
  name = "terraform-test"
  network_interface = {
    network = "default"
  }
  network_interface = {
    mac = "52:54:00:a9:f5:17"
  }
}
 ``

`Network not found` error in refreshing state

Reproducer:

• create a managed network (sample main.tf)
• terraform apply
• delete the network manually: virsh net-destroy mynetwork; virsh net-undefine mynetwork
• terraform apply

Expected result: the network is re-created.

Actual result:

libvirt_network.network: Refreshing state... (ID: ...)
Error refreshing state: 1 error(s) occurred:

* libvirt_network.network: [Code-43] [Domain-19] Network not found: no network with matching uuid...

Workaround: terraform state rm libvirt_network.network

Add support for managed storage pools

Currently storage pools are left to be managed manually, it would be good to have them managed through the plugin.

Establish IP addresses once they are assigned

When using managed networks we can currently specify the IP address manually, or let dnsmasq get an IP from the DHCP pool automatically. The problem with the automatic assignation is that it does survive reboots, so clusters can get messed up easily.

We could add an option (or do it automatically) for a) getting the IP address assigned by dnsmasq to the VM and b) add the IP-MAC mapping to the the managed network definition.

Establish a version

Right now this project is totally unversioned. I think it's time for us to start to tag each significant release. This will make packaging easier on our side, will make it easier to understand which version is in place and will be more respectful of the human civilization.

I propose to start with version 0.1.0.

@inercia, @dmacvicar, @moio: what do you think about it?

Define multiple disks

Is this supported? I attempted this but it defines both disks as sda.

XML error: target 'sda' duplicated for disk sources

I'm trying to attach an Ubuntu cloud image and a config drive to a guest.

Crash on destroy

Version Reports:

Distro version of host:

openSUSE 42.2

Terraform Version Report

Terraform v0.8.8

Libvirt version

2.0.0

terraform-provider-libvirt plugin version (git-hash)

git log

---

Description of Issue/Question

Got this panic:

panic: sync: unlock of unlocked mutex
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: goroutine 45 [running]:
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: panic(0x9f8a40, 0xc420325760)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/runtime/panic.go:500 +0x1a1
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: sync.(*Mutex).Unlock(0xc420437218)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/sync/mutex.go:111 +0x104
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.LibVirtPoolSync.ReleaseLock(0xc42016ef00, 0x0, 0xc4203256f8, 0x8)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/pool_sync.go:48 +0xe2
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.RemoveVolume(0xc420208be0, 0xc4202fcf00, 0x2c, 0x0, 0x0)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/utils.go:101 +0x6ff
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.resourceLibvirtVolumeDelete(0xc4203170e0, 0x9d9840, 0xc420208be8, 0xc4202539c8, 0x414cd4)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/resource_libvirt_volume.go:352 +0x7b
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc420377260, 0xc4202b6aa0, 0xc420313860, 0x9d9840, 0xc420208be8, 0x1, 0xc420426600, 0x18)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:137 +0x3eb
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc4203773e0, 0xc4202b6a50, 0xc4202b6aa0, 0xc420313860, 0x0, 0x0, 0x0)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:212 +0x9b
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc4203fe040, 0xc4203137e0, 0xc420325720, 0x0, 0x0)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/home/alvaro/Development/go/src/github.com/dmacvicar/terraform-provider-libvirt/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:488 +0x57
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: reflect.Value.call(0xc420284480, 0xc4202089e0, 0x13, 0xae7eb4, 0x4, 0xc420253ef0, 0x3, 0x3, 0xc42002eec8, 0x552e69, ...)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/reflect/value.go:434 +0x5c8
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: reflect.Value.Call(0xc420284480, 0xc4202089e0, 0x13, 0xc420253ef0, 0x3, 0x3, 0xc420131338, 0xc42016f740, 0xc42002eeec)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/reflect/value.go:302 +0xa4
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: net/rpc.(*service).call(0xc4203fd980, 0xc4203fd940, 0xc42041a4d8, 0xc42020b780, 0xc42043c200, 0x9db940, 0xc4203137e0, 0x16, 0x9db980, 0xc420325720, ...)
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/net/rpc/server.go:383 +0x148
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: created by net/rpc.(*Server).ServeCodec
2017/04/08 01:19:40 [DEBUG] plugin: terraform-provider-libvirt: 	/usr/lib64/go/src/net/rpc/server.go:477 +0x421
2017/04/08 01:19:40 [DEBUG] plugin: /home/alvaro/Applications/terraform-provider-libvirt: plugin process exited

Allow to specify disks/storage for each domain

Libvirt allows for almost unlimited flexibility when configuring a virtual machine.

For the context of terraform, I think it makes sense to support two use cases:

• An empty disk, used for storage
  • in this case terraform will create the volume and save the id/name, and destroy it together with the domain.
• A disk image (eg. OS), specifying a storage pool and a volume name
  • Additionally, specifying the image as a remote HTTP url and automatically upload it to the pool may be a useful feature for the terraform conect
  • In this case, terraform should not destroy it, as this is seen as a template
  • Another idea would be to create a volume like in the first case, using the template as the backing file.

Some drafting:

disk {
  source = "http.//opensuse.org/tumbleweed.qcow2"
  pool = "default"
}

disk {
  pool = "default"
  name = "foobar.qcow"
}

disk {
   size = 2400 # MiB
}

Provider ignores "running" parameter for domain

Version Reports:

Distro version of host:

Ubuntu 14.04

Terraform Version Report

v0.8.6

Libvirt version

1.3.1

terraform-provider-libvirt plugin version (git-hash)

a07e21b

---

Description of Issue/Question

Despite "running = false" domain always running.

Setup

provider "libvirt" {
  uri = "qemu+ssh://root@host/system"
}

resource "libvirt_domain" "rs-removeme" {
  name = "rs-removeme"
  vcpu = 1
  memory = 4096
  running = false
}

Steps to Reproduce Issue

1. Create file with state above
2. terraform apply. See log

Expected result:
VM is not running

Actual result:
VM is running.

---

Additional Infos:

No Apparmor

Offering a helping hand

Firstly, i just want to say i've been looking for a qemu provider for Terraform for some time now and you are a life saver! Now, i've never programmed in golang before so i wouldnt be of much help there, but if you can think of any other way i can help with this project(ie documentation, tutorials etc, by all means let me know. I would be more then happy to help.

Add volume_attachment resources or equivalent

I want to create a libvirt domain with a variable number of attached volumes (case in point is one main disk always present, and one optional data disk).

Currently this is not possible in any way I tried. Although the disk attribute looks like a map it cannot be passed parameters (the following does not work):

resource "libvirt_domain" "domain" {
disk = ["${var.disks}"]

Where var.disks is a map with a volume_id key and a correct id value. In fact not even the following works:

disk = ["${map("volume_id", "123")}"]

We get the following error on 0.7.7:

Error reading config for libvirt_domain[domain]: At 31:11: root.disk[0]: not an object type for map (*ast.LiteralType)

It would be great to have a way around this, eg:

• having a disk_ids plain list of ids instead of a list of maps/objects
• having a separate resource to associate disks to instances, similar to aws_volume_attachment
• being able to use a plain map for the existing disk variable.

Thanks in advance!!!

Allow to define networks resources

No mockup yet...

resource "libvirt_network" "mynetwork" {

}

Asignation of cloud-init iso fails

When cloud-init iso is created, the volume is defined with Type "qcow2" instead of "raw".
I've created a PR #59 to solve this issue.

Hostname or IP as an output of domain

A domain's IP/hostname might not be known before the resource is created, for example if bridged networking is used with an external DHCP.

libvirt knows though - possibly through the qemu guest agent - can we make this an output of the domain resource for later use?

Cloudinit requires local_hostname field

but the docs are mentioning it as optional.

* libvirt_cloudinit.test_ci: "local_hostname": required field is not set

Crash on apply of libvirt_domain

Version Reports:

Distro version of host:

Ubuntu 16.04.2

Terraform Version Report

Terraform v0.9.1

Libvirt version

1.3.1

terraform-provider-libvirt plugin version (git-hash)

commit a07e21b74726d9ce443ac9332417c169b2ccd708
Author: Eamonn O'Toole <[email protected]>
Date:   Thu Mar 16 16:12:22 2017 +0000

---

Description of Issue/Question

On applying the libvirt_domain config terraform crashes (but the VM is actually running fine, including networking):

2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: panic: runtime error: invalid memory address or nil pointer dereference
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x52c6ad]
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: goroutine 64 [running]:
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/libvirt-go.(*VirDomain).ListAllInterfaceAddresses(0xc420273370, 0x0, 0x1, 0xc4203c43f0, 0x0, 0x0, 0xb694d0)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/dmacvicar/libvirt-go/domain.go:767 +0x35d
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.getDomainInterfaces(0xc420273370, 0x44, 0xc42039e230, 0x1, 0x1, 0x1)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/resource_libvirt_domain.go:983 +0x113
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.waitForNetworkAddresses(0xc4202803c0, 0x1, 0x1, 0x7ff20c001220, 0x1, 0x10b70e0)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/resource_libvirt_domain.go:909 +0x279
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/dmacvicar/terraform-provider-libvirt/libvirt.resourceLibvirtDomainCreate(0xc42035a540, 0xa28a40, 0xc4202dab58, 0x0, 0x0)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/dmacvicar/terraform-provider-libvirt/libvirt/resource_libvirt_domain.go:485 +0x385e
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc4202ce840, 0xc420266960, 0xc4203846c0, 0xa28a40, 0xc4202dab58, 0x1, 0x28, 0xc420379b90)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/hashicorp/terraform/helper/schema/resource.go:186 +0x48d
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc42025bab0, 0xc420266910, 0xc420266960, 0xc4203846c0, 0x7ff23d8422c0, 0x0, 0x0)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/hashicorp/terraform/helper/schema/provider.go:242 +0x9b
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc4203067c0, 0xc420384140, 0xc420381610, 0x0, 0x0)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /home/frederik/src/_go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:488 +0x57
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: reflect.Value.call(0xc4202563c0, 0xc420272840, 0x13, 0xb3fe76, 0x4, 0xc420265f20, 0x3, 0x3, 0xb694d0, 0xc420207ee8, ...)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /usr/local/go/src/reflect/value.go:434 +0x91f
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: reflect.Value.Call(0xc4202563c0, 0xc420272840, 0x13, 0xc420207f20, 0x3, 0x3, 0xc420207f0c, 0x180001, 0x500000000)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /usr/local/go/src/reflect/value.go:302 +0xa4
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: net/rpc.(*service).call(0xc4203013c0, 0xc420301380, 0xc42030a9c0, 0xc420308080, 0xc42034c480, 0xa29a40, 0xc420384140, 0x16, 0xa29a80, 0xc420381610, ...)
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /usr/local/go/src/net/rpc/server.go:387 +0x144
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt: created by net/rpc.(*Server).ServeCodec
2017/03/25 18:49:30 [DEBUG] plugin: terraform-provider-libvirt:         /usr/local/go/src/net/rpc/server.go:481 +0x404
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalWriteState
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalApplyProvisioners
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalIf
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalWriteState
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalWriteDiff
2017/03/25 18:49:30 [DEBUG] root: eval: *terraform.EvalApplyPost
2017/03/25 18:49:30 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* libvirt_domain.test01: unexpected EOF
2017/03/25 18:49:30 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* libvirt_domain.test01: unexpected EOF
2017/03/25 18:49:30 [TRACE] [walkApply] Exiting eval tree: libvirt_domain.test01
2017/03/25 18:49:30 [DEBUG] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"

Setup

provider "libvirt" {
    uri = "qemu:///system"
}

resource "libvirt_volume" "mydisk" {
  name             = "terraform-test1.img"
  pool             = "default"
  base_volume_name = "centos-7.3_vagrant_box_image_0.img"
}

resource "libvirt_domain" "test01" {
  name    = "terraform_test01"
  vcpu    = 2
  memory  = 1024
  running = "false"

  disk {
    volume_id = "${libvirt_volume.mydisk.id}"
    scsi      = "yes"
  }

  network_interface {
    network_name   = "default"
    hostname       = "test1"
    mac            = "AA:BB:CC:11:22:22"
    wait_for_lease = 1
  }
}

Steps to Reproduce Issue

terraform plan
terraform apply

See attached crash log.

---

Additional Infos:

Apparmor is disabled, I do have a firewall (ufw) enabled on both the host, but this is working fine as I can actually login using SSH to the newly started VM and see that it has networking operational just fine.

storage pool bug encountered

I got this issue ( 1 time out of 3 maybe) when setting up 2 VMs

• libvirt_volume.centos7_vm_disk: Error refreshing pool for volume: [Code-1] [Domain-18] internal error: pool 'default' has asynchronous jobs running.

I wonder if you could make the provider a little more robust, with a few retries maybe, this is really a temporary failure I think and it seems related to an old libvirt bug.

Use virDomainLookupByUUIDString for the resource ID

Needs to be added to https://github.com/rgbkrk/libvirt-go first.

• rgbkrk/libvirt-go#61

Instead of using the hypervisor ID.

• It is not hypervisor specific, which may be a good idea in order to move the connection URI to the resource itself in the future and allow migration of resources.
• No need to convert int <-> string

excluded vendor directory

Is there any reason the vendor directory is excluded? Most, if not all, go projects include the vendor directory in their projects. This would also make packaging easier on some distributions.

"count" example out of sync

Needs to be updated with changes in commit 8496660

Provider ignores

Version Reports:

Distro version of host:

openSUSE 42.2/ Centos7/ Ubuntu..

Terraform Version Report

(Provided by running terraform -v.)

Libvirt version

virsh --version

terraform-provider-libvirt plugin version (git-hash)

git log

---

Description of Issue/Question

Setup

(Please provide the full main.tf file for reproducing the issue (Be sure to remove sensitive info)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

---

Additional Infos:

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration?
Have you tried to reproduce the issue without them enabled?

libvirt_volume.main_disk: Can't retrieve volume xxx

How-to-reproduce:

clone sumaform
Take the main.tf-libvirt-testing example.

Terraform get
Terraform apply

run it 3/4 times. ( 2 times will fail with error :

libvirt_volume.main_disk: Can't retrieve volume  sles12sp1

This issue block the provissioning and for production or dayly work is annoyng, because we have to reapply terraform apply.

I cannot sadly determine the cause of this issue, the disk created is there but terraform fails to retrieve it.
It's a race condition. I'm quite sure that the sumaform is not the root cause of the problem.

For additional logs, info, ping me

update version of terraform dependency

In order for this provider to work with the newest release of Terraform, the dependency in the vendor.yml needs to be updated to v0.7.0. I will create a PR which resolves this issue, if that's OK.

Add progress to volume download/upload

May be using https://gowalker.org/github.com/mitchellh/ioprogress

remote libvirt URI, terraform cant' find storage pool 'default'

How to reproduce

1. Using the https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/examples/count/libvirt.tf

Change the uri to a remote kvm/libvirt server.,

provider 'libvirt'{
  uri = "qemu+tcp://10.*.*.*/system"
}

terraform plan

libvirt_domain.domain.0
    disk.#:           "1"
    disk.0.volume_id: "${element(libvirt_volume.volume.*.id, count.index)}"
    memory:           "512"
    name:             "domain-0"
    running:          "true"
    vcpu:             "1"

+ libvirt_domain.domain.1
    disk.#:           "1"
    disk.0.volume_id: "${element(libvirt_volume.volume.*.id, count.index)}"
    memory:           "512"
    name:             "domain-1"
    running:          "true"
    vcpu:             "1"

+ libvirt_domain.domain.2
    disk.#:           "1"
    disk.0.volume_id: "${element(libvirt_volume.volume.*.id, count.index)}"
    memory:           "512"
    name:             "domain-2"
    running:          "true"
    vcpu:             "1"

+ libvirt_domain.domain.3
    disk.#:           "1"
    disk.0.volume_id: "${element(libvirt_volume.volume.*.id, count.index)}"
    memory:           "512"
    name:             "domain-3"
    running:          "true"
    vcpu:             "1"

+ libvirt_volume.opensuse_leap
    name:   "opensuse_leap"
    pool:   "default"
    size:   "<computed>"
    source: "http://download.opensuse.org/repositories/Cloud:/Images:/Leap_42.1/images/openSUSE-Leap-42.1-OpenStack.x86_64.qcow2"

+ libvirt_volume.volume.0
    base_volume_id: "${libvirt_volume.opensuse_leap.id}"
    name:           "volume-0"
    pool:           "default"
    size:           "<computed>"

+ libvirt_volume.volume.1
    base_volume_id: "${libvirt_volume.opensuse_leap.id}"
    name:           "volume-1"
    pool:           "default"
    size:           "<computed>"

+ libvirt_volume.volume.2
    base_volume_id: "${libvirt_volume.opensuse_leap.id}"
    name:           "volume-2"
    pool:           "default"
    size:           "<computed>"

+ libvirt_volume.volume.3
    base_volume_id: "${libvirt_volume.opensuse_leap.id}"
    name:           "volume-3"
    pool:           "default"
    size:           "<computed>"

terraform apply

terraform apply
libvirt_volume.opensuse_leap: Creating...
  name:   "" => "opensuse_leap"
  pool:   "" => "default"
  size:   "" => "<computed>"
  source: "" => "http://download.opensuse.org/repositories/Cloud:/Images:/Leap_42.1/images/openSUSE-Leap-42.1-OpenStack.x86_64.qcow2"
Error applying plan:

1 error(s) occurred:

* libvirt_volume.opensuse_leap: can't find storage pool 'default'

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Remarks:

so i think the problem is with the disk/volume in a remote scenario URI.

I did a test, with the "minimal example" here :
https://github.com/dmacvicar/terraform-provider-libvirt#using-the-provider

and it worked, i mean terraform could create a domain on the remote machine.

I can access to the remote machine via virsh, so there is no network/authentification problem

Include provisioner remote-exec

There will bee a good option to enhance next version with terraform provisioner remote-exec.
I think will be easy than configure multiple options in libvirt_cloudinit resource.

volume source does not accept dots in file name.

I don't know if the problem is in qemu, but using:

// Than you may define a template, this file is not modified
resource "libvirt_volume" "template7" {
pool = "ram"
name = "template7"
source = "http://localhost:8000/centos7.std.base"
}

it gave me:

qemu-img: /home/vmstore/ramdisk/centos7vm1_mydisk: Could not open backing file: Could not open '/home/vmstore/ramdisk/centos7.base': No such file or directory

some part of the filename was stripped out.

Creating multiple domains with count fails due to spice port already in use

Terraform file: https://gist.github.com/adamcstephens/7e56423ccceb3e1f3c0d808eb7cb8c59

(Ignore my serialconsole parameter, you'll be getting a PR for that. :)

Error applying plan:

3 error(s) occurred:

* libvirt_domain.centos.3: Error creating libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: ((null):9070): Spice-Warning **: reds.c:2490:reds_init_socket: listen: Address already in use
2016-11-11T01:54:21.893813Z qemu-system-x86_64: failed to initialize spice server
* libvirt_domain.centos.0: Error creating libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: annel/target/domain-3-centos0/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -object rng-random,id=objrng0,filename=/dev/random -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x7 -msg timestamp=on
Domain id=3 is tainted: high-privileges
((null):9075): Spice-Warning **: reds.c:2490:reds_init_socket: listen: Address already in use
2016-11-11T01:54:21.893813Z qemu-system-x86_64: failed to initialize spice server
* libvirt_domain.centos.2: Error creating libvirt domain: [Code-1] [Domain-10] internal error: process exited while connecting to monitor: x/store/qiic7ibvl75rrdmrq5d8w202165j25i9-dmidecode-3.0/sbin:/nix/store/bs6640nvs7wxn8qrg14snwzds7mzh7x9-dnsmasq-2.76/sbin:/nix/store/4pjlmmhm7cq9h53xbmm96y77xpibsk7b-ebtables-2.0.10-4/sbin:/nix/store/ldcxp98nkgnjzqs4ki7p7mxrhdgrzsjn-qemu-x86-only-2.7.0/sbin:/nix/store/8rn45r9ndfq5h7mx58r35p2szky5ja6n-coreutils-8.25/sbin:/nix/store/gjm2wjh8m72ch3ikiznn4v40h9mvpfks-findutils-4.6.0/sbin:/nix/store/h9aqgpgspgjhygj63wpncfzvz5ys851n-gnugrep-2.25/sbin:/nix/store/1awgi4ba6ymlzib2l7bxa13i17nq27d0-gnused-4.2.2/sbin:/nix/store/inbr0qixfmq0js9pwm8n0bhqad865mab-systemd-231/sbin QEMU_AUDIO_DRV=spice /nix/store/ldcxp98nkgnjzqs4ki7p7mxrhdgrzsjn-qemu-x86-only-2.7.0/bin/qemu-kvm -name guest=centos2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-centos2/master-key.aes -machine pc-i440fx-2.7,accel=kvm,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 3c8941a6-37e6-45ad-a617-f5efab2188d4 -no-user-config -nodefa

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

RFC: revisit how disks are added to domains

The current state

Right now to add a disk to a domain we must:

1. declare a volume
2. create a disk entry linked with the volume

The problems

These are the minor issues/annoyances:

• accidentally sharing a volume between different domains
• using count on a domain introduces quite some complexity: you have to use count also inside of the volume definition and you have to be careful when writing the link (the terraform element syntax is not exactly user friendly)
• a lot of typing: for each domain you have to define a dedicated volume

The biggest issue from my POV is that the volume life cycle is kinda disconnected from the one of the domain.

Doesn't play well when rebuilding a domain

When you change a domain in a way that it forces its rebuild the following actions will happen:

• the domain will be deleted
• the volume won't be deleted
• a new domain will be created, the old volume will be used by it
• the new domain is started
• the provisioners of the domain are re-executed

Bad things can happen during the last step because most of the times the provisioners assume they are running on a "vanilla" system while they already ran against that volume.

Doesn't play well with changes

Imagine a domain has already been provisioned and the user changes its disks. The next apply will unlink the old volumes from the domain and link the freshly created ones.

This has two problems:

• there's no way to trigger the execution of the provisioners
• the old volume is left inside of the poll until the user removes its definition (which might be inside of another .tf file) or when the whole infra is brought to ashes

At the same time changing a linked volume won't cause the domain provisioners to be executed again. This is IMHO a pretty big issue.

How to fix that

Terraform original design

Terraform was designed to handle cloud workloads. In this context there are two types of storage devices: ephemeral and persistent ones.

Each instance has always an ephemeral disk. The life cycle of this disk is coupled with the instance. A change that forces the rebuilding of the instance will cause the disk to be deleted and the new instance will start with a clean slate.

The persistent storage is usually called volume and can be attached to an instance. This is not really tied to the life cycle of an instance, hence it can be used to store really important data that has to be preserved between instance rebuilds.

The proposal

There's no way to "taint" a linked resource. We cannot mark all the volumes linked to a domain as to be deleted whenever the domain is rebuilt. I think the feature is missing because it would not make sense inside of the original context for which Terraform has been designed.
We basically have to implement ephemeral and persistent storage with libvirt.

The persistent one is the simplest, we just have to use the volume resource to implement that. However we should introduce new directives to allow the user to create a volume from nothing (like by specifying just the size). This can be useful also outside of the context of this issue.

The ephemeral one is a bit tricky, especially from the user point of view. We could extend the disk type to have the following attributes:

• pool: target pool to use
• source: image to use. This is going to be uploaded to the pool and used directly (no linked disk underneath)
• base_volume_id: link against an already defined volume
• base_volume_pool: look for a base volume inside of that pool
• base_volume_name: link against a volume that is identified by a name instead of an internal terraform ID

Defining a domain would be something like that:

resource "libvirt_domain" "chameleon" {
  name = "chameleon"
  disk {
    source = "opensuse-leap.qcow2"
  }
}

This would cause the opensuse-leap.qcow2 image to be uploaded to the default pool as a volume named: <domain name>.img, hence chameleon.img.

Everything is still handled as a volume from the libvirt POV, however the domain resource has full responsibility at managing this volume. It's just a matter of extending the create, delete and update functions of the domain resource.

The usability question

Ephemeral and persistent volumes have still to be added as disks to a domain. How can we differentiate between what is going to survive domain rebuilds and what is going to be nuked every time?

We could define persistent volumes in this way:

disk {
  volume_id = "${...}"
}```

And ephemeral ones in this way:
```json
disk {
   base_volume_{id|name} = "..."
}

However I think it could lead to confusion among users...

Feedback

What do you think about the whole thing?

If you are interested I can share a POF I made (it supports create and delete of ephemeral disks created with source).

output ip address for interfaces (as output variables)

When a VM is started, an output variable with every interface ip addresses should be set.

Requires some work in libvirt-go, being done here: https://github.com/dmacvicar/libvirt-go/tree/domain_iface_addr in order to add virDomainInterfaceAddresses.

The current idea is to set an option per network interface that allows to wait until that interface gets an ip address. Something like:

resource "libvirt_domain" "domain1" {
  name = "domain1"
  disk {
    volume_id = "${libvirt_volume.myvol.id}"
  }
  network_interface {
    network = "default"
    wait_for_address = true
  }

If wait_for_address is true then terraform apply would wait until that interface gets an address and then set a variable network_interface.0.ip_v4.0 to x.x.x.x.

@fisabelle @johnavp1989 input appreciated.

[Network] Error defining libvirt network: [Code-9] [Domain-19] operation failed: network 'terraform-network' already exists

Description:

When the network managed by terraform exists already, and you want to spawn machines with terraform-libvirt, you get an error:

Error applying plan:

1 error(s) occurred:

* libvirt_network.network: Error defining libvirt network: [Code-9] [Domain-19] operation failed: network 'terraform-network' already exists with uuid 7a8d2cae-9e60-4863-977c-2f7d07486b36 -   <network>
      <name>terraform-network</name>
      <domain name="tf.local"></domain>
      <bridge stp="on"></bridge>
      <forward mode="nat">
          <nat>
              <port start="1024" end="65535"></port>
          </nat>
      </forward>
      <ip address="192.168.127.1" prefix="24" family="ipv4">
          <dhcp>
              <range start="192.168.127.2" end="192.168.127.254"></range>
          </dhcp>
      </ip>
  </network>

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

machines are are setup correctly and network is ok

How to reproduce:

Take a main.tf from sumaform: https://github.com/MalloZup/sumaform/blob/master/main.tf.libvirt-testing.example

Cannot unlink file '/var/lib/libvirt/images/XXXXX': Permission denied

• Probably related to https://bugzilla.redhat.com/show_bug.cgi?id=1293804
• Need to be tested in different libvirt versions (eg. Leap)
• For now we use workaround https://bugzilla.redhat.com/show_bug.cgi?id=1293804#c12
  • Does not solve the problem but it makes it happen less often (eg. "sometimes")