nbering / terraform-inventory

An Ansible dynamic inventory script to pair with nbering/terraform-provider-ansible.

Terraform Inventory

An Ansible dynamic inventory script to process Terraform state and return Ansible host data from Terraform Provider for Ansible host resources. See the Terraform Provider for it's own installation and use.

Usage

Copy the terraform.py script file to a location on your system. Ansible's own documentation suggests the location /etc/ansible/terraform.py, but the particular location does not matter to the script. Ensure it has executable permissions (chmod +x /etc/ansible/terraform.py).

With your Ansible playbook and Terraform configuration in the same directory, run Ansible with the -i flag to set the inventory source.

$ ansible-playbook -i /etc/ansible/terraform.py playbook.yml

Environment Variables

ANSIBLE_TF_BIN

Override the path to the Terraform command executable. This is useful if you have multiple copies or versions installed and need to specify a specific binary. The inventory script runs the terraform state pull command to fetch the Terraform state, so that remote state will be fetched seemlessly regardless of the backend configuration.

ANSIBLE_TF_DIR

Set the working directory for the terraform command when the scripts shells out to it. This is useful if you keep your terraform and ansible configuration in separate directories. Defaults to using the current working directory.

ANSIBLE_TF_WS_NAME

Sets the workspace for the terraform command when the scripts shells out to it, defaults to default workspace - if you don't use workspaces this is the one you'll be using.

License

Licensed for use under the MIT License.

terraform-inventory's Issues

Get ansible_ resources from multiple Remote state

Having terraform configuration in different directories/account for secuirity/sepration o duty or something else, can be useful terraform inventory allow to read states from other directories out of the main as current or defined by ANSIBLE_TF_DIR.

E.g. gathering ansible configuration from multimple remote state (multiple directoies in ANSIBLE_TF_DIR), to work like multiple terraform_remote_state in terraform:

data "terraform_remote_state" "dir1" {...}
data "terraform_remote_state" "dir2" {...}
...
data "terraform_remote_state" "dirn" {...}

Every directory will have its backend with aws connection config ti allow this design, probably.

Document ad-hoc test framework

I've got a little test framework going that just has sample terraform configurations, with state files checked-in. Along with them, there as a JSON file that represents the expected output from terraform.py for that case.

There should be a CONTRIBUTING.md file to explain how to use all that.

local exec fails to parse terraform.py

There seems to be a problem parsing the terraform.py inside a local-exec provisioner in Terraform

Taking this as an example trigger in terraform to run ansible

resource "null_resource" "jenkins_ansible" {
  // count = 2 ### "${length(list(var.jenkins_private_ip))}"
  triggers {
    // jenkins_ips = "${var.jenkins_private_ip[count.index]}"
    jenkins_ips = "${var.jenkins_private_ip}"
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command = "cd ${var.ansible_dir}; ANSIBLE_TF_DIR=${var.ANSIBLE_TF_DIR} ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook --inventory-file=${var.ansible_dir}/terraform.py jenkins/master.yml"
  }
}

gives me this output

[WARNING]:  * Failed to parse /Users/moi/ansible/sandbox-
test/terraform.py with script plugin: Inventory script
(/Users/moi/ansible/sandbox-test/terraform.py) had an execution
error: 2018/03/14 13:44:52 [INFO] Terraform version: 0.11.3
3802b14260603f90c7a1faf55994dcc8933e2069 2018/03/14 13:44:52 [INFO] Go runtime
version: go1.9.1 2018/03/14 13:44:52 [INFO] CLI args: []string{"terraform",
"state", "pull", "-input=false"} 2018/03/14 13:44:52 [DEBUG] Attempting to open
CLI config file: /Users/moi/.terraformrc 2018/03/14 13:44:52
[DEBUG] File doesn't exist, but doesn't need to. Ignoring. 2018/03/14 13:44:52
[INFO] CLI command args: []string{"state", "pull", "-input=false"} 2018/03/14
13:44:52 [DEBUG] command: loading backend config file:
/Users/moi/foo/git-dev/idam/idam-iac/devops-
idam-static-dev 2018/03/14 13:44:52 [WARN] BackendOpts.Config not set, but
config found 2018/03/14 13:44:52 [TRACE] Preserving existing state lineage
"a34ef702-b45c-49ee-ab06-985baa37f7eb" 2018/03/14 13:44:52 [TRACE] Preserving
existing state lineage "a34ef702-b45c-49ee-ab06-985baa37f7eb" 2018/03/14
13:44:52 [INFO] Building AWS region structure 2018/03/14 13:44:52 [INFO]
Building AWS auth structure 2018/03/14 13:44:52 [INFO] Setting AWS metadata API
timeout to 100ms 2018/03/14 13:44:53 [INFO] Ignoring AWS metadata API endpoint
at default location as it doesn't return any instance-id 2018/03/14 13:44:53
[INFO] AWS Auth provider used: "SharedCredentialsProvider" 2018/03/14 13:44:53
[INFO] Initializing DeviceFarm SDK connection 2018/03/14 13:44:53 [DEBUG]
Trying to get account ID via iam:GetUser 2018/03/14 13:44:54 [INFO] command:
backend initialized: *s3.Backend 2018/03/14 13:44:54 [DEBUG] checking for
provider in "." 2018/03/14 13:44:54 [DEBUG] checking for provider in
"/terraform" 2018/03/14 13:44:54 [DEBUG] checking for provider in
".terraform/plugins/darwin_amd64" 2018/03/14 13:44:54 [DEBUG] found provider
"terraform-provider-aws_v1.11.0_x4" 2018/03/14 13:44:54 [DEBUG] found provider
"terraform-provider-null_v1.0.0_x4" 2018/03/14 13:44:54 [DEBUG] checking for
provider in "/Users/moi/.terraform.d/plugins" 2018/03/14 13:44:54
[DEBUG] checking for provider in
"/Users/moi/.terraform.d/plugins/darwin_amd64" 2018/03/14
13:44:54 [DEBUG] found provider "terraform-provider-ansible_v0.0.2" 2018/03/14
13:44:54 [DEBUG] found valid plugin: "aws", "1.11.0",
"/Users/moi/foo/git-dev/idam/idam-iac
/devops-idam-static-dev/.terraform/plugins/darwin_amd64/terraform-provider-
aws_v1.11.0_x4" 2018/03/14 13:44:54 [DEBUG] found valid plugin: "null",
"1.0.0", "/Users/moi/foo/git-dev/idam/idam-
iac/devops-idam-static-dev/.terraform/plugins/darwin_amd64/terraform-provider-
null_v1.0.0_x4" 2018/03/14 13:44:54 [DEBUG] found valid plugin: "ansible",
"0.0.2", "/Users/moi/.terraform.d/plugins/darwin_amd64/terraform-
provider-ansible_v0.0.2" 2018/03/14 13:44:54 [DEBUG] checking for provisioner
in "." 2018/03/14 13:44:54 [DEBUG] checking for provisioner in "/terraform"
2018/03/14 13:44:54 [DEBUG] checking for provisioner in
".terraform/plugins/darwin_amd64" 2018/03/14 13:44:54 [DEBUG] checking for
provisioner in "/Users/moi/.terraform.d/plugins" 2018/03/14
13:44:54 [DEBUG] checking for provisioner in
"/Users/moi/.terraform.d/plugins/darwin_amd64" 2018/03/14
13:44:54 [INFO] command: backend *s3.Backend is not enhanced, wrapping in local
2018/03/14 13:44:54 [TRACE] Preserving existing state lineage "294a1e0c-
cb5d-4365-90e6-0f6a9cecb4fc" 2018/03/14 13:44:54 [TRACE] Preserving existing
state lineage "294a1e0c-cb5d-4365-90e6-0f6a9cecb4fc" 2018/03/14 13:44:54
[TRACE] Preserving existing state lineage "294a1e0c-
cb5d-4365-90e6-0f6a9cecb4fc" 2018/03/14 13:44:54 [DEBUG] plugin: waiting for
all plugin processes to complete...
 [WARNING]:  * Failed to parse /Users/moi/ansible/sandbox-
test/terraform.py with ini plugin: /Users/moi/ansible/sandbox-
test/terraform.py:3: Expected key=value host variable assignment, got: json
 [WARNING]: Unable to parse /Users/moi/ansible/sandbox-
test/terraform.py as an inventory source
 [WARNING]: No inventory was parsed, only implicit localhost is available
 [WARNING]: provided hosts list is empty, only localhost is available. Note
that the implicit localhost does not match 'all'
 [WARNING]: Could not match supplied host pattern, ignoring: jenkins_hosts

when the same command is run in the terminal it works just fine

$ cd /Users/moi/ansible/sandbox-test; ANSIBLE_TF_DIR=/Users/moi/foo/git-dev/idam/idam-iac/devops-idam-static-dev ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook --inventory-file=/Users/moi/ansible/sandbox-test/terraform.py jenkins/master.yml

Should it be used like that within Terraform?

As I am very close to creating a separate application to that is triggered on remote state callback that will then trigger the ansible scripts, and have a feeling it could be a more logical way to separate the concern and domains.

Wondering if anyone else came across the same issue.

Script -> plugin possibility

Since ansible 2.4 they've started suggesting dynamic inventory scripts be changed to inventory plugins. Is this something you've considered doing?

I'm using your TF provider plugin and it seems great so far. I'll likely take a stab at making an ansible inventory plugin though. I'm trying to avoid having to set an active directory or env variables. The new plugin structure is similar to the way scripts worked except now you can actually pass in a parameter, which allows the env variables to go away. The possibility of multiple, prioritized plugins also makes it nice for having an option between an approach like yours that actually calls terraform state and an approach that just reads a remote state file directly. E.g. ansible-playbook -i tfstate:s3://my-state-bucket/my-app/terraform.tfstate my_playbook.yml vs ansible-playbook -i tfdir:~/my_repo/tf/my_app my_playbook.yml.

I'd be interested in your opinion. If you like, I'll let you know how my attempt goes with making the plugin.

Issue with attributes for ansible group

I have the following being reported in my terminal:

ANSIBLE_TF_WS_NAME=staging ./terraform-inventory.py 
Traceback (most recent call last):
  File "./terraform-inventory.py", line 400, in _main
    inventory.add_resource(resource)
  File "./terraform-inventory.py", line 243, in add_resource
    self.add_group_resource(resource)
  File "./terraform-inventory.py", line 216, in add_group_resource
    groupname = resource.read_attr("inventory_group_name")
  File "./terraform-inventory.py", line 184, in read_attr
    return self._raw_attributes().get(key, None)
  File "./terraform-inventory.py", line 189, in _raw_attributes
    return self.source_json["attributes"]
KeyError: 'attributes'

This is using a 'remote' terraform backend, Terraform v0.12.0. The state says it is version 4. Here's an example of the state:

{
  "version": 4,
  "terraform_version": "0.12.0",
  "serial": 2,
  "lineage": "1a478ae2-6647-8ebd-0815-46b948b7b22e",
  "outputs": {
    "infra_vpn_static_ip": {
      "value": "139.59.203.99",
      "type": "string"
    }
  },
  "resources": [
    {
      "mode": "managed",
      "type": "ansible_group",
      "name": "database",
      "provider": "provider.ansible",
      "instances": [
        {
          "schema_version": 0,
          "attributes_flat": {
            "id": "database",
            "inventory_group_name": "database"
          }
        }
      ]
    }
}

I can see 'attributes_flat' but not 'attributes'. I've tried changing a value here and there to see if it worked, but I'm not a python dev so I can't see how the data is structured at line 189 to see what the key should be.

I'll keep digging though.

Better merge support for overlapping variable keys

Add vars merging with better understanding of configuration semantics. ie. ansible_host_var should probably override ansible_host by default since an individual var is more specific.

Could also support a priority property on the resource to allow fine-grained control of how vars are merged together.

json encoding support for python 3.9

encoding argument deprecated in json library since python 3.1 and it will be removed in python 3.9.

https://docs.python.org/3/library/json.html#json.loads

json.loads(s, *, cls=None, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, object_pairs_hook=None, **kw)
Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using this conversion table.
The other arguments have the same meaning as in load(), except encoding which is ignored and deprecated since Python 3.1.
If the data being deserialized is not a valid JSON document, a JSONDecodeError will be raised.
Deprecated since version 3.1, will be removed in version 3.9: encoding keyword argument.
Changed in version 3.6: s can now be of type bytes or bytearray. The input encoding should be UTF-8, UTF-16 or UTF-32.

Unable to parse

Hello,
Terraform 0.12.17
Ansible 2.9.6

ansible-playbook -i /etc/ansible/terraform.py play/test.yml
 [WARNING]:  * Failed to parse /etc/ansible/terraform.py with script plugin: Inventory script (/etc/ansible/terraform.py) had an execution error: Traceback (most recent call last):
File "/etc/ansible/terraform.py", line 395, in _main     tfstate = TerraformState(_execute_shell())   File "/etc/ansible/terraform.py", line 390, in _execute_shell     return
json.loads(out_cmd, encoding=encoding)   File "/usr/lib/python3.6/json/__init__.py", line 354, in loads     return _default_decoder.decode(s)   File
"/usr/lib/python3.6/json/decoder.py", line 339, in decode     obj, end = self.raw_decode(s, idx=_w(s, 0).end())   File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

 [WARNING]:  * Failed to parse /etc/ansible/terraform.py with ini plugin: /etc/ansible/terraform.py:3: Error parsing host definition ''''': No closing quotation

 [WARNING]: Unable to parse /etc/ansible/terraform.py as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Already do chmod +x

Part of my tfstate:

      "mode": "managed",
      "type": "ansible_host",
      "name": "wordpress01",
      "provider": "provider.ansible",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "groups": [
              "wordpress"
            ],
            "id": "wordpress01",
            "inventory_hostname": "wordpress01",
            "variable_priority": 50,
            "vars": {
              "ansible_connection": "lxd"
            }
          }

Where is my error?

workspaces not supported

Hi

I'm using the Gitlab Remote State Backend for my terraform state.
While executing the terraform.py I get the following error:

$ ansible-playbook -i /etc/ansible/terraform.py ../ansible/playbook.yml [WARNING]: * Failed to parse /etc/ansible/terraform.py with script plugin: Inventory script (/etc/ansible/terraform.py) had an execution error: workspaces not supported [WARNING]: * Failed to parse /etc/ansible/terraform.py with ini plugin: /etc/ansible/terraform.py:2: Error parsing host definition ''''': No closing quotation [WARNING]: Unable to parse /etc/ansible/terraform.py as an inventory source [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

chmod +x was executed on the file and I also tried:
ANSIBLE_TF_DIR=/path/to/terraform/workdir/ ansible -i terraform.py ...

what is my error?

Add a note about project structure for Terraform + Ansible to README

I've made some assumptions in the design that prefer a project were Terraform and Ansible configuration are in the same directory, because that's how I organize things.

I've also provided a mechanism for configuring the script to have a separate Terraform directory. It's documented under Environment Variables, but something clearer under Usage, or a separate section on project structure would probably be better.

Upstream?

Are you considering to upstream this as an inventory plugin into Ansible, so that it is available to every user of Ansible right from the start?

Import groups resources from tfstate.

Tracking along nbering/terraform-provider-ansible#8, add comprehension for groups, include child groups and group variables.

fill running playbook

running ansible-playbook -i ../../allinlab-terraform/terraform.py common.yml
returns error

[WARNING]: * Failed to parse /Users/elevin/git/allinlab-terraform/terraform.py with script plugin: Inventory script
(/Users/elevin/git/allinlab-terraform/terraform.py) had an execution error: Empty state (no state)

[WARNING]: * Failed to parse /Users/elevin/git/allinlab-terraform/terraform.py with ini plugin: /Users/elevin/git/allinlab-
terraform/terraform.py:3: Expected key=value host variable assignment, got: json

[WARNING]: Unable to parse /Users/elevin/git/allinlab-terraform/terraform.py as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

I attached the Json state
state.txt

hostvars is emply when run ./terraform.py

Hello. Thanks for terraform-inventory.

terraform -version
Terraform v0.11.14

apply terraform from repo https://github.com/yandex-cloud/examples/tree/master/active-directory

Run terraform.py

./terraform.py
{
  "_meta": {
    "hostvars": {}
  }

terraform -version
Terraform v0.12.29

Simple terraform main.tf:

  resources {
    cores  = 2
    memory = 4
  }

  boot_disk {
    initialize_params {
      image_id = "fd87va5cc00gaq2f5qfb"
    }
  }

  network_interface {
    subnet_id = yandex_vpc_subnet.subnet-1.id
    nat       = true
  }

  metadata = {
    ssh-keys = "ubuntu:${file("~/.ssh/id_rsa.pub")}"
  }
}

resource "yandex_vpc_network" "network-1" {
  name = "network1"
}

resource "yandex_vpc_subnet" "subnet-1" {
  name           = "subnet1"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.network-1.id
  v4_cidr_blocks = ["192.168.10.0/24"]
}

output "internal_ip_address_vm_1" {
  value = yandex_compute_instance.vm-1.network_interface.0.ip_address
}

output "internal_ip_address_vm_2" {
  value = yandex_compute_instance.vm-2.network_interface.0.ip_address
}


output "external_ip_address_vm_1" {
  value = yandex_compute_instance.vm-1.network_interface.0.nat_ip_address
}

output "external_ip_address_vm_2" {
  value = yandex_compute_instance.vm-2.network_interface.0.nat_ip_address
}

Run terraform.py

 ./terraform.py
{
  "_meta": {
    "hostvars": {}
  }

Add support for individual vars resources.

Work being done in nbering/terraform-provider-ansible#18 will add ansible_host_var and ansible_group_var resources.

Values set by these resources should be merged with the ansible_host and ansible_group variables respectively.

Failed to parse with script plugin error post macOS upgrade (Terraform remote state?)

Hi,

I'm aware that there is an open issue (#21) but given it's a couple of years old I'm not sure the workarounds mentioned in there are still correct. In addition, I've started seeing this error quite recently after upgrading to macOS Monterey so I'm not sure if it's some type of broken dependency.

I'm seeing the following error when I try to run any ansible command with the Inventory script:

❯ ansible -i /Users/balaji/Applications/terraform.py online -m ping
/usr/local/Cellar/[email protected]/5.7.1/libexec/lib/python3.10/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[WARNING]:  * Failed to parse /Users/balaji/Applications/terraform.py with script plugin: Inventory script
(/Users/balaji/Applications/terraform.py) had an execution error: env: python: No such file or directory
[WARNING]:  * Failed to parse /Users/balaji/Applications/terraform.py with ini plugin: /Users/balaji/Applications/terraform.py:3: Error
parsing host definition ''''': No closing quotation
[WARNING]: Unable to parse /Users/balaji/Applications/terraform.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: online

Terraform state is stored remotely, but I'm able to retrieve Terraform state in the same directory without any error:

❯ terraform state list
data.aws_ssm_parameter.proxmox_api_address
data.aws_ssm_parameter.proxmox_login
data.aws_ssm_parameter.proxmox_login_secret
ansible_host.airconnect
ansible_host.devdesktop
... [output snipped]

Is there some way to debug this issue further? I've tried adding the ANSIBLE_TF_DIR variable when running ansible commands but that doesn't seem to change anything.