picatz / terraform-google-nomad Goto Github PK

View Code? Open in Web Editor NEW

78.0 6.0 16.0 380 KB

📗 Terraform Module for Nomad clusters with Consul on GCP

Home Page: https://registry.terraform.io/modules/picatz/nomad/google

License: MIT License

HCL 66.35% Shell 13.05% Go 11.11% Makefile 9.50%

nomad terraform gcp mtls ssh acls consul-connect packer consul

terraform-google-nomad's People

Contributors

Stargazers

Watchers

Forkers

pljeff allisson rbramwell crankycoder karimjedda strixbe gjpin dhung-hashicorp momania tgross riskiwah alfred-hq knanao erkankesik sagarrakshe

terraform-google-nomad's Issues

Use managed instance groups for server and client deployments

It would probably be a good idea to use managed instance groups for Nomad server and client deployments in the future with the compute_instance_group_manager Terraform resource. GoogleCloudPlatform/terraform-google-managed-instance-group looks like a good starting point.

Need to look into how that works, implement it, and then document how to trigger rolling updates for Nomad.

Add an option to deploy Consul with Nomad

Nomad's integration with Consul is really great, and there should be an option to enable deploying a secure-by-default Consul cluster to make it easy to get started.

Why is the nomad server configuration using a temporary folder for data?

I see that this line configures the server to use a temporary path for data, is it intended, or should it also use /nomad/data?

Side question: if we run both a binary with the server config and another binary with the client config, on the same server, both pointing to the same data dir, does it cause issues?

Interpolation-only Expressions Are Deprecated

$ terraform validate .

Warning: Interpolation-only expressions are deprecated

  on modules/open-port/firewall.tf line 6, in resource "google_compute_firewall" "open_port":
   6:     protocol = "${var.protocol}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

(and 4 more similar warnings elsewhere)

Dependabot can't parse your acl_tokens.tf

Dependabot couldn't parse the acl_tokens.tf found at /acl_tokens.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:27: Unknown token: 2:27 IDENT uuid

View the update logs.

Auto-update tutorial example module version upon release

Run a GitHub Actions workflow upon a new version release to keep the tutorial content up-to-date.

on:
  release:
    types: [published]

Falco Installation Broken

$ make packer/build
2021-05-30T11:51:05-04:00: ==> client: E: Failed to fetch https://dl.bintray.com/falcosecurity/deb/dists/stable/InRelease  403  Forbidden [IP: 54.185.54.139 443]
2021-05-30T11:51:05-04:00: ==> client: E: The repository 'https://dl.bintray.com/falcosecurity/deb stable InRelease' is not signed.

Hardcoded Project Name Usage in VM Module Image Var

Hardcoded project names were accidentally included included. The image variable should use the format function with the hardcoded packer image name, and the required project variable for this module.

Run nomad as non root user

Hi!

I was referencing this repo for deploying a similar stack on AWS. I came across install_nomad.sh#L27 and I think this is a typo. Should it not be owned by nomad user instead of root user?

sudo chown --recursive nomad:nomad /nomad

Does not work on `darwin_arm64`

$ terraform init 
╷
│ Error: Incompatible provider version
│ 
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.
│ 
│ Provider releases are separate from Terraform CLI releases, so not all providers are available for all platforms. Other versions of
│ this provider may have different platforms supported.
╵

The last release was two years ago.

Install the Stackdriver monitoring agent

To start getting the builtin observability features working.

curl -sSO https://dl.google.com/cloudagents/add-monitoring-agent-repo.sh && sudo bash add-monitoring-agent-repo.sh --also-install && sudo service stackdriver-agent start

Likely need to add a new scope to the vm module:

https://www.googleapis.com/auth/monitoring.write

Prevent Access to the Metadata Service from Container Workloads using iptables

Related to #19: Nomad client nodes should have this iptables rule included to prevent access to the metadata service API:

$ iptables --insert FORWARD 1 --in-interface nomad --destination 169.254.169.254/32 --jump DROP

It should also be persisted across restarts.

Make the Packer Template User-friendly

Right now the Packer template has a hard-coded project_id. To make this more friendly to use, the template needs to be updated to include a user variable, and the README documentation should be adjusted.

Prevent Plaintext Secrets in the Compute Metadata Service

As of v2.0.0/#18 , this module deploys a Consul cluster in tandem with the Nomad cluster. Moreover, it uses the metadata service to perform the majority of the dynamic server configuration, and exposes many secrets to malicious/compromised workloads on Nomad client instances.

All secrets should be removed from the metadata service, or at least not stored in plaintext.

Stackdriver Logging agent error, VMs have insufficient authentication scopes

All nodes within the cluster are configured with the google-fluentd Stackdriver Logging agent. When looking at the logs from the agent, I'm getting the following errors:

$ cat /var/log/google-fluentd/google-fluentd.log 
2020-07-12 18:15:56 +0000 [warn]: Failed to extract log entry errors from the error details: "Request had insufficient authentication scopes.". error_class=JSON::ParserError error="String"
...

☝️ This also means the VM instance logs aren't available in the GCP console.

I think I need to expand the service_accounts.scopes for VMs to also include a logging scope to fix this error.

Dependabot can't parse your locals.tf

Dependabot couldn't parse the locals.tf found at /locals.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:18: Unknown token: 2:18 IDENT base64encode

View the update logs.

Install Falco

Falco can be included to enable runtime security monitoring by default.

Dependabot can't parse your load_balancer.tf

Dependabot couldn't parse the load_balancer.tf found at /load_balancer.tf.

The error Dependabot encountered was:

unable to parse HCL: At 3:20: Unknown token: 3:20 IDENT var.region

View the update logs.

Investigate providing a Firecracker task driver option

It would be interesting to provide an option to enable running a Firecracker task driver since GCP supports nested virtualization.

Supporting Documentation

Use Ops Agent

GCP now recommends using the new Ops Agent for monitoring and logging.

Permission error when setting up a server

I have this error when trying to setup the consul server, not sure how to add the required permissions?

Cannot discover address: cluster=LAN address="provider=gce project_name=<project>  tag_value=server" error="discover-gce: googleapi: Error 403: Required 'compute.zones.list' permission for 'projects/<project-name>'"

Dependabot can't parse your bucket.tf

Dependabot couldn't parse the bucket.tf found at /bucket.tf.

The error Dependabot encountered was:

unable to parse HCL: At 2:19: Unknown token: 2:19 IDENT format

View the update logs.

Generate mTLS certificates dynamically with Terraform

Instead of generating the mTLS certificates with cfssl before using Packer during the creation of the images, it'd be great to just dynamically create those certificates when I spin up the cluster using Terraform's tls provider.

Allow more configuration of Consul integration

Currently, there is no way to really tune the Consul integration outside of enabling/disabling Consul ACLs and the default policy.

terraform-google-nomad/vars.tf

Lines 138 to 148 in 8c61d66

 variable "consul_acls_enabled" { 

 type = bool 

 default = true 

 description = "If ACLs should be enabled for the Consul cluster." 

 } 

 variable "consul_acls_default_policy" { 

 type = string 

 default = "deny" 

 description = "The default policy to use for Consul ACLs (allow/deny)." 

 }

But there are many options available. These should be exposed as Terraform variables with secure defaults.

terraform-google-nomad/packer/configs/nomad/server.hcl

Lines 45 to 53 in 8c61d66

 consul { 

 ssl = true 

 verify_ssl = true 

 address = "127.0.0.1:8501" 

 ca_file = "/consul/config/consul-ca.pem" 

 cert_file = "/consul/config/server.pem" 

 key_file = "/consul/config/server-key.pem" 

 token = "{CONSUL-TOKEN}" 

 }

Extra important ones to consider would be allow_unathenticated and share_ssl. Consider disabling these by default with adjustments to documentation and examples.

Consul configuration defines unused options in server and client

Client agents do not need to enable connect, this is only used on servers:

Enabling Connect requires changing the configuration of only your Consul servers (not client agents).

terraform-google-nomad/packer/configs/consul/client.hcl

Lines 30 to 32 in 0db159e

 connect { 

 enabled = true 

 }

Servers do not need to enable the gRPC port, this is only used on clients:

terraform-google-nomad/packer/configs/consul/server.hcl

Line 17 in 0db159e

grpc = 8502

There might be others, but these should definitely be removed.

Add a Vault integration option

Support an option to deploy Vault with github.com/picatz/terraform-google-vault

Minor typo, should be "sensitive"

terraform-google-nomad/ssh-mtls-terminating-proxy.go

Line 24 in b7ea9c7

Senstive bool `json:"senstive"`

Investigate running Rootless Docker Daemon on Nomad clients

🤦 Following up on a1a3f29, when rootless and experimental were set to true, the Docker Daemon still seemed to be running as root. So, I seem to be missing something. Need to further investigate out that works, and then probably reimplement it.

Errors and other alerts when following the tutorial steps

In step 4, right after executing the command 'terraform plan...' I have the output below, please how can I solve this problem?

Terraform v0.12.28

provider.google v3.65.0
provider.local v2.1.0
provider.random v3.1.0
provider.template v2.2.0
provider.tls v3.1.0

Warning: registry.terraform.io: This version of Terraform has an outdated GPG key and is unable to verify new provider releases. Please upgrade Terraform to at least 0.12.31 to receive new provider updates. For details see: https://discuss.hashicorp.com/t/hcsec-2021-12-codecov-security-event-and-hashicorp-gpg-key-exposure/23512