alphagov / terraform-provider-concourse

A terraform provider for Concourse

terraform-provider-concourse

What

A terraform provider for concourse

Why

fly is an amazing tool, but configuration using scripts running fly is not ideal.

Prerequisites

Install go, and terraform.

How to install and use

make install

How to build and test for development

make
make integration-tests

Example terraform

Create a provider (using target from fly)

provider "concourse" {
  target = "target_name"
}

Create a provider (using a local username and password)

Note: this is not basic authentication

provider "concourse" {
  url  = "https://wings.pivotal.io"
  team = "main"

  username = "localuser"
  password = "very-secure-password"
}

Look up all teams

data "concourse_teams" "teams" {
}

output "team_names" {
  value = data.concourse_teams.teams.names
}

Look up a team

data "concourse_team" "my_team" {
  team_name = "main"
}

output "my_team_name" {
  value = data.concourse_team.my_team.team_name
}

output "my_team_owners" {
  value = data.concourse_team.my_team.owners
}

output "my_team_members" {
  value = data.concourse_team.my_team.members
}

output "my_team_pipeline_operators" {
  value = data.concourse_team.my_team.pipeline_operators
}

output "my_team_viewers" {
  value = data.concourse_team.my_team.viewers
}

Look up a pipeline

data "concourse_pipeline" "my_pipeline" {
  team_name     = "main"
  pipeline_name = "pipeline"
}

output "my_pipeline_team_name" {
  value = data.concourse_pipeline.my_pipeline.team_name
}

output "my_pipeline_pipeline_name" {
  value = data.concourse_pipeline.my_pipeline.pipeline_name
}

output "my_pipeline_is_exposed" {
  value = data.concourse_pipeline.my_pipeline.is_exposed
}

output "my_pipeline_is_paused" {
  value = data.concourse_pipeline.my_pipeline.is_paused
}

output "my_pipeline_json" {
  value = data.concourse_pipeline.my_pipeline.json
}

output "my_pipeline_yaml" {
  value = data.concourse_pipeline.my_pipeline.yaml
}

Create a team

Supports owners, members, pipeline_operators, and viewers.

Specify users and groups by prefixing the strings:

• user:
• group:

resource "concourse_team" "my_team" {
  team_name = "my-team"

  owners = [
    "group:github:org-name",
    "group:github:org-name:team-name",
    "user:github:tlwr",
  ]

  viewers = [
    "user:github:samrees"
  ]
}

Create a pipeline

resource "concourse_pipeline" "my_pipeline" {
  team_name     = "main"
  pipeline_name = "my-pipeline"

  is_exposed = true
  is_paused  = true

  pipeline_config        = file("pipeline-config.yml")
  pipeline_config_format = "yaml"
}

# OR

resource "concourse_pipeline" "my_pipeline" {
  team_name     = "main"
  pipeline_name = "my-pipeline"

  is_exposed = true
  is_paused  = true

  pipeline_config        = file("pipeline-config.json")
  pipeline_config_format = "json"
}

terraform-provider-concourse's Issues

External diff for `pipeline_config` not detected

Since pipeline_config is not read from the API during the Read operation, a diff for it is not picked up when manually changing the pipeline configuration using fly.

Reproducing:

1. Create pipeline with Terraform, in this example named my-test-pipeline.
2. fly get-pipeline -p my-test-pipeline > ~/testing.yaml
3. Edit testing.yaml with your changes
4. fly set-pipeline -p my-test-pipeline -c ~/test.yaml
5. Run terraform plan or terraform refresh and see no differences detected

Support login provider for user in LDAP

Hello,

Provider does not work for users in LDAP scope.

Missing license

This project doesn't have a license. I'd like to reuse/help, but I can't unless it's licensed in a corporate friendly way (BSD, apache 2.0, MIT, etc). Would it be possible to add a license?

Thanks!

Unable to build (Linux)

I tried to build this provider according to the readme instructions but get the following errors:

glide install

[INFO]  Lock file (glide.lock) does not exist. Performing update.
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/ghodss/yaml
[INFO]  --> Fetching updates for golang.org/x/oauth2
[INFO]  --> Fetching updates for github.com/concourse/concourse
[INFO]  --> Fetching updates for github.com/onsi/ginkgo
[INFO]  --> Fetching updates for github.com/hashicorp/terraform
[INFO]  --> Fetching updates for github.com/onsi/gomega
[INFO]  Resolving imports
[INFO]  --> Fetching updates for github.com/hashicorp/go-hclog
[INFO]  --> Fetching updates for github.com/hashicorp/go-plugin
[INFO]  --> Fetching updates for github.com/zclconf/go-cty
[INFO]  --> Fetching updates for google.golang.org/grpc
[INFO]  --> Fetching updates for github.com/agext/levenshtein
[INFO]  --> Fetching updates for github.com/hashicorp/errwrap
[INFO]  --> Fetching updates for github.com/hashicorp/go-multierror
[INFO]  --> Fetching updates for github.com/hashicorp/go-uuid
[INFO]  --> Fetching updates for github.com/hashicorp/go-version
[INFO]  --> Fetching updates for github.com/hashicorp/hcl
[INFO]  --> Fetching updates for github.com/mitchellh/copystructure
[INFO]  --> Fetching updates for github.com/mitchellh/reflectwalk
[INFO]  --> Fetching updates for github.com/davecgh/go-spew
[INFO]  --> Fetching updates for github.com/hashicorp/logutils
[INFO]  --> Fetching updates for github.com/mitchellh/colorstring
[INFO]  --> Fetching updates for github.com/mitchellh/mapstructure
[INFO]  --> Fetching updates for github.com/jessevdk/go-flags
[INFO]  --> Fetching updates for github.com/tedsuo/rata
[INFO]  --> Fetching updates for golang.org/x/crypto
[INFO]  --> Fetching updates for sigs.k8s.io/yaml
[INFO]  --> Fetching updates for github.com/peterhellberg/link
[INFO]  --> Fetching updates for github.com/cppforlife/go-semi-semantic
[INFO]  --> Fetching updates for gopkg.in/yaml.v2
[INFO]  --> Fetching updates for github.com/fatih/color
[INFO]  --> Fetching updates for github.com/mattn/go-colorable
[INFO]  --> Fetching updates for github.com/mattn/go-isatty
[INFO]  --> Fetching updates for github.com/hashicorp/yamux
[INFO]  --> Fetching updates for github.com/mitchellh/go-testing-interface
[INFO]  --> Fetching updates for github.com/oklog/run
[INFO]  --> Fetching updates for golang.org/x/net
[INFO]  --> Fetching updates for github.com/golang/protobuf
[INFO]  --> Fetching updates for github.com/hashicorp/go-getter
[INFO]  --> Fetching updates for github.com/hashicorp/terraform-svchost
[INFO]  --> Fetching updates for github.com/mitchellh/cli
[INFO]  --> Fetching updates for golang.org/x/text
[INFO]  --> Fetching updates for github.com/vmihailenco/msgpack
[ERROR] Error scanning github.com/hashicorp/hcl/v2: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2
[ERROR] Error scanning github.com/hashicorp/hcl/v2/hcldec: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/hcldec
[ERROR] Error scanning github.com/hashicorp/hcl/v2/hclsyntax: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/hclsyntax
[INFO]  --> Fetching updates for github.com/blang/semver
[INFO]  --> Fetching updates for github.com/hashicorp/hil
[INFO]  --> Fetching updates for github.com/mitchellh/hashstructure
[INFO]  --> Fetching updates for github.com/spf13/afero
[INFO]  --> Fetching updates for github.com/hashicorp/go-cleanhttp
[INFO]  --> Fetching updates for github.com/zclconf/go-cty-yaml
[INFO]  --> Fetching updates for github.com/mitchellh/go-wordwrap
[INFO]  --> Fetching updates for github.com/hashicorp/terraform-config-inspect
[INFO]  --> Fetching updates for github.com/bmizerany/pat
[INFO]  --> Fetching updates for github.com/vito/go-sse
[INFO]  --> Fetching updates for github.com/google/jsonapi
[INFO]  --> Fetching updates for github.com/hpcloud/tail
[INFO]  --> Fetching updates for golang.org/x/sys
[INFO]  --> Fetching updates for google.golang.org/appengine
[INFO]  --> Fetching updates for google.golang.org/genproto
[INFO]  --> Fetching updates for cloud.google.com/go
[INFO]  --> Fetching updates for github.com/aws/aws-sdk-go
[INFO]  --> Fetching updates for github.com/bgentry/go-netrc
[INFO]  --> Fetching updates for github.com/hashicorp/go-safetemp
[INFO]  --> Fetching updates for github.com/mitchellh/go-homedir
[INFO]  --> Fetching updates for github.com/ulikunitz/xz
[INFO]  --> Fetching updates for google.golang.org/api
[INFO]  --> Fetching updates for github.com/armon/go-radix
[INFO]  --> Fetching updates for github.com/bgentry/speakeasy
[INFO]  --> Fetching updates for github.com/posener/complete
[INFO]  --> Fetching updates for github.com/vmihailenco/tagparser
[ERROR] Error scanning github.com/hashicorp/hcl/v2/gohcl: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/gohcl
[ERROR] Error scanning github.com/hashicorp/hcl/v2/hclparse: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/hclparse
[ERROR] Error scanning github.com/hashicorp/hcl/v2/ext/typeexpr: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/ext/typeexpr
[ERROR] Error scanning github.com/hashicorp/hcl/v2/ext/dynblock: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/ext/dynblock
[INFO]  --> Fetching updates for github.com/apparentlymart/go-cidr
[INFO]  --> Fetching updates for github.com/bmatcuk/doublestar
[INFO]  --> Fetching updates for github.com/google/uuid
[INFO]  --> Fetching updates for github.com/apparentlymart/go-textseg
[ERROR] Error scanning github.com/hashicorp/hcl/v2/hcled: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-hashicorp-hcl/v2/hcled
[INFO]  --> Fetching updates for github.com/hashicorp/hcl2
[INFO]  --> Fetching crypto/ed25519
[WARN]  Unable to checkout crypto/ed25519
[ERROR] Error looking for crypto/ed25519: Cannot detect VCS
[INFO]  Found Godeps.json file in /home/tabby/.glide/cache/src/https-github.com-hpcloud-tail
[INFO]  --> Parsing Godeps metadata...
[INFO]  --> Fetching updates for gopkg.in/tomb.v1
[INFO]  --> Setting version for gopkg.in/tomb.v1 to c131134a1947e9afd9cecfe11f4c6dff0732ae58.
[INFO]  --> Fetching updates for github.com/googleapis/gax-go
[ERROR] Error scanning github.com/vmihailenco/msgpack/v4/codes: cannot find package "." in:
        /home/tabby/.glide/cache/src/https-github.com-vmihailenco-msgpack/v4/codes
[INFO]  --> Fetching updates for gopkg.in/fsnotify/fsnotify.v1
[INFO]  --> Fetching updates for github.com/jstemmer/go-junit-report
[INFO]  --> Fetching updates for golang.org/x/exp
[INFO]  --> Fetching updates for golang.org/x/lint
[INFO]  --> Fetching updates for golang.org/x/tools
[INFO]  --> Fetching updates for honnef.co/go/tools
[INFO]  --> Fetching updates for go.opencensus.io
[INFO]  --> Fetching updates for github.com/jmespath/go-jmespath
[INFO]  --> Fetching updates for github.com/google/go-cmp
[INFO]  --> Fetching updates for github.com/golang/groupcache
[INFO]  --> Fetching updates for github.com/BurntSushi/toml
[ERROR] Failed to retrieve a list of dependencies: Error resolving imports

go build

pkg/provider/team.go:8:2: cannot find package "github.com/concourse/concourse/atc" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/concourse/concourse/atc (vendor tree)
        /usr/lib/go/src/github.com/concourse/concourse/atc (from $GOROOT)
        /home/tabby/go/src/github.com/concourse/concourse/atc (from $GOPATH)
pkg/provider/config.go:6:2: cannot find package "github.com/concourse/concourse/fly/rc" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/concourse/concourse/fly/rc (vendor tree)
        /usr/lib/go/src/github.com/concourse/concourse/fly/rc (from $GOROOT)
        /home/tabby/go/src/github.com/concourse/concourse/fly/rc (from $GOPATH)
pkg/client/client.go:7:2: cannot find package "github.com/concourse/concourse/go-concourse/concourse" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/concourse/concourse/go-concourse/concourse (vendor tree)
        /usr/lib/go/src/github.com/concourse/concourse/go-concourse/concourse (from $GOROOT)
        /home/tabby/go/src/github.com/concourse/concourse/go-concourse/concourse (from $GOPATH)
pkg/provider/util.go:5:2: cannot find package "github.com/ghodss/yaml" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/ghodss/yaml (vendor tree)
        /usr/lib/go/src/github.com/ghodss/yaml (from $GOROOT)
        /home/tabby/go/src/github.com/ghodss/yaml (from $GOPATH)
pkg/provider/config.go:8:2: cannot find package "github.com/hashicorp/terraform/helper/schema" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/hashicorp/terraform/helper/schema (vendor tree)
        /usr/lib/go/src/github.com/hashicorp/terraform/helper/schema (from $GOROOT)
        /home/tabby/go/src/github.com/hashicorp/terraform/helper/schema (from $GOPATH)
main.go:4:2: cannot find package "github.com/hashicorp/terraform/plugin" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/hashicorp/terraform/plugin (vendor tree)
        /usr/lib/go/src/github.com/hashicorp/terraform/plugin (from $GOROOT)
        /home/tabby/go/src/github.com/hashicorp/terraform/plugin (from $GOPATH)
main.go:5:2: cannot find package "github.com/hashicorp/terraform/terraform" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/github.com/hashicorp/terraform/terraform (vendor tree)
        /usr/lib/go/src/github.com/hashicorp/terraform/terraform (from $GOROOT)
        /home/tabby/go/src/github.com/hashicorp/terraform/terraform (from $GOPATH)
pkg/client/client.go:8:2: cannot find package "golang.org/x/oauth2" in any of:
        /home/tabby/go/src/github.com/alphagov/terraform-provider-concourse/vendor/golang.org/x/oauth2 (vendor tree)
        /usr/lib/go/src/golang.org/x/oauth2 (from $GOROOT)
        /home/tabby/go/src/golang.org/x/oauth2 (from $GOPATH)

Output of go env:

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tabby/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/tabby/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="x86_64-pc-linux-gnu-gcc"
CXX="x86_64-pc-linux-gnu-g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build376764370=/tmp/go-build -gno-record-gcc-switches"

Error with dynamic across value interpolation

Hi,

I'm having problems with using a varabilized input for across values. I've put together an example below to replicate.

If I replace values: ((.:config.versions)) with something hardcoded, everything runs fine as expected. However, with a variable it's not happy and breaks, but manually setting the pipeline through fly works.

Any help on this would be greatly appreciated, thanks!

Example config:

pipeline.yml:

jobs:
  - name: hello
    plan:
      - { load_var: config, file: config.yml, format: "yml" }
      - across:
          - var: version
            values: ((.:config.versions))
        task: say-hello
        config:
          platform: linux
          image_resource: { type: registry-image , source: { repository: "busybox" }}
          run: { path: echo , args: [ "Hello", ((.:version)),  "!" ] }

config.yml:

  versions:
    - a
    - b
    - c

main.tf:

terraform {
  required_providers {
    concourse = {
      source = "terraform-provider-concourse/concourse"
      version = "8.0.1"
    }
  }
}

provider "concourse" {
  target = "<target>"
}

resource "concourse_pipeline" "pipeline_test" {
  is_exposed = false
  is_paused = false
  pipeline_config = file("pipeline.yml")
  pipeline_config_format = "yaml"
  pipeline_name = "pipeline-test"
  team_name = "<team>"
}

I can run the following: fly -t <target> set-pipeline --pipeline pipeline-test --config pipeline.yml fine, but with either an apply through Terraform before the pipeline is applied (note, it will apply fine and show up in Concourse, but still show the error) or any subsequent applies afterwards, results in:

│ Error: Error reading pipeline pipeline-test from team '<team>': Error looking up pipeline pipeline-test within team '<team>': malformed across step: json: cannot unmarshal string into Go struct field AcrossStep.across of type []interface {}

Add possibility to archive pipelines instead of destroying them

Hello!

I was wondering if the pipelines could be archived, instead of being destroyed, when the resource is removed from the configuration. At least for me that would be a great option, because then I'm able to view old builds in my organisation, after I've removed them from my configuration. (E.g. for Post mortem scenarios)

Another benefit for me would be that the penalty for accidentally removing a resource wouldn't be as high. In that case, I could just revert the change and restore the build history thereby.

Since this idea is a rather incisive change I would suggest a opt-in configuration.

My preferred way to configure this would be a flag on the provider itself, say archive_instead_of_delete. Another possibility would be another flag in the pipeline-resource, say is_archived. However I don't like the idea of another flag because it may make most other settings obsolete in the case of an archived pipeline. It furthermore has a huge potential for conflicts, therefore I would prefer a flag on the provider.

A possible provider configuration may look like this then:

provider "concourse" {
  url    = "https://example.com"
  target = "base"

  # This defaults to false
  archive_instead_of_delete = true
}

I could contribute a PR for this as well and like to know if this is desired.

What do you think about this, do you have any experience in this topic, any concerns or another suggestion?

Thanks

Update - Fly version

There's not been a release for a while, any chance that we could get a new version of the fly CLI? I believe currently it's ~7.5.1.

Publish provider

Hi!

First of all, thanks for the provider. Using it and it works a treat. It'd be nice, especially for local users, if it didn't require manually building and discovering. There's some docs about setting up this officially and terraform .13 will support 3rd party providers better.

https://www.terraform.io/docs/registry/providers/publishing.html

Are you open to this? Obviously I can PR an action to build and release, but I can't be providing registry accounts and signing keys.

Support for Concourse 6.2.0.

Concourse since 6.1.0 has backward incompatible changes and this plugin needs to be adjusted for recent version.

PR: #10

feature request: add variables

Hello!

Would it be please possible to add a variables option for set-pipeline (same as --var)

fly set-pipeline --help
Usage:
  fly [OPTIONS] set-pipeline [set-pipeline-OPTIONS]

Application Options:
  -t, --target=                     Concourse target name
  -v, --version                     Print the version of Fly and exit
      --verbose                     Print API requests and responses
      --print-table-headers         Print table headers even for redirected output

Help Options:
  -h, --help                        Show this help message

[set-pipeline command options]
      -n, --non-interactive         Skips interactions, uses default values
          --no-color                Disable color output
      -d, --dry-run                 Run a set pipeline step but in dry-run mode
          --check-creds             Validate credential variables against credential manager
      -p, --pipeline=               Pipeline to configure
      -c, --config=                 Pipeline configuration file, "-" stands for stdi
-->>  -v, --var=[NAME=STRING]       Specify a string value to set for a variable in the pipeline
      -y, --yaml-var=[NAME=YAML]    Specify a YAML value to set for a variable in the pipeline
      -l, --load-vars-from=         Variable flag that can be used for filling in template values in configuration from a YAML file
          --team=                   Name of the team to which the pipeline belongs, if different from the target default

Thanks!

How can I set OIDC User or Group?

I set github outh but our team need to change it to keycloak using oidc.

below team resource. guide me to set owners arg. please...

resource "concourse_team" "my_team" {

  team_name = "myteam"
  owners = [

  ]
}

Feature - Pipeline template `((variables))` interpolation

Thank you for developing and publishing this provider.

requested feature context

Pipeline template variable syntax: https://concourse-ci.org/vars.html#var-syntax

Processes of performing interpolation of variables is handled within fly
https://github.com/concourse/concourse/blob/4189a7ed0b920f1de9d7c468caf40047126383b9/fly/commands/set_pipeline.go#L104

and therefore is missing from this provider, meaning to many/most users the provider cannot act as a drop-in replacement for fly in the trivial usecase and is only suitable where either

• the pipeline config contains no "Static Vars" (https://concourse-ci.org/vars.html#static-vars)
• some other 3rd party performs pipeline template generation

❯ fly sp -h
Usage:
✂️
[set-pipeline command options]
✂️
      -v, --var=[NAME=STRING]       Specify a string value to set for a variable in the pipeline
      -l, --load-vars-from=         Variable flag that can be used for filling in template values in configuration from a YAML file
✂️

high-level suggested implementation

Resources affected

• concourse_pipeline - addition of attribute vars to concourse_pipeline resource.
  • could be map(string) using TF config directly (ex 1) or accept YAML/JSON encoded input similar to pipeline_config attr (ex 2)

resource "concourse_pipeline" "my_pipeline" {
  team_name     = "main"
  pipeline_name = "my-pipeline"

  is_exposed = true
  is_paused  = true

  pipeline_config        = file("pipeline-config.yml")
  pipeline_config_format = "yaml"
  vars = { # 👋 
    "what_day_is_it" = "tuesday",
  }
}

or 2.

resource "concourse_pipeline" "my_pipeline" {
  team_name     = "main"
  pipeline_name = "my-pipeline"

  is_exposed = true
  is_paused  = true

  pipeline_config        = file("pipeline-config.yml")
  pipeline_config_format = "yaml"
  vars                   = file("pipeline-vars.yml") # 👋 
}

• could consider importing templatehelpers mod or related from fly project to get templating logic for free

Running with terraform 0.13?

Hi all,
First of all thanks for the project!
How can I run it with terraform 0.13?
I did make install and add provider to my config.tf, but terraform init produces error:

Error: Failed to install provider

Error while installing hashicorp/concourse: provider registry
[registry.terraform.io](http://registry.terraform.io/) does not have a provider named
[registry.terraform.io/hashicorp/concourse](http://registry.terraform.io/hashicorp/concourse)

I believe I need to add required_providers.concourse.source, but what should I write to source?

Thanks in advance

feature: Validate pipeline config

We should be able to use the ATC configvalidate module¹ in the same fashion that the validate helpers for fly² does.

Footnotes

1. https://github.com/concourse/concourse/blob/master/atc/configvalidate/validate.go ↩

2. https://github.com/concourse/concourse/blob/master/fly/commands/internal/validatepipelinehelpers/validate.go ↩