hashicorp / terraform-aws-terraform-enterprise

A Terraform module for deploying Terraform Enterprise on AWS.

Terraform Enterprise AWS Module

IMPORTANT: You are viewing a beta version of the official module to install Terraform Enterprise. This new version is incompatible with earlier versions, and it is not currently meant for production use. Please contact your Customer Success Manager for details before using.

This is a Terraform module for provisioning a Terraform Enterprise Cluster on AWS. Terraform Enterprise is our self-hosted distribution of Terraform Cloud. It offers enterprises a private instance of the Terraform Cloud application, with no resource limits and with additional enterprise-grade architectural features like audit logging and SAML single sign-on.

About This Module

This module will install Terraform Enterprise on AWS according to the HashiCorp Reference Architecture. This module is intended to be used by practitioners seeking a Terraform Enterprise installation which requires minimal configuration in the AWS cloud.

As the goal for this main module is to provide a drop-in solution for installing Terraform Enterprise via the Golden Path, it leverages AWS native solutions such as Route 53 and a vanilla AWS-supplied base AMI. We have provided guidance and limited examples for other use cases.

Pre-requisites

This module is intended to run in an AWS account with minimal preparation, however it does have the following pre-requisites:

Terraform version >= 0.14

This module requires Terraform version 0.14 or greater to be installed on the running machine.

Credentials / Permissions

AWS Services Used

• AWS Identity & Access Management (IAM)
• AWS Key Management System (KMS)
• Amazon RDS (Postgres)
• Amazon EC2
• Amazon Elastic Loadbalancing (ALB)
• Amazon Certificate Manager (ACM)
• Amazon Route53
• Amazon Elasticache (Redis)
• Amazon VPC
• Amazon S3
• [OPTIONAL] Amazon Secrets Manager

Public Hosted Zone

If you are managing DNS via AWS Route53 the hosted zone entry is created automatically as part of your domain management.

If you're managing DNS outside of Route53, please see the documentation on creating a hosted zone for a subdomain, which you will need to do for the subdomain you are planning to use for your Terraform Enterprise installation. To create this hosted zone with Terraform, use the aws_route53_zone resource.

ACM Certificate

Certificate validation can take up two hours, causing timeouts during module apply if the cert is generated as one of the resources contained in the module. For that reason, once the hosted zone has been created, the certificate must be created or imported into ACM. To create or import manually, see the AWS ACM certificate documentation. To create or manage certificates with Terraform, we recommend the official ACM module in the Terraform Registry.

Note: This module has been tested in the following AWS regions:

• us-east-1
• eu-west-1
• eu-west-2

How to Use This Module

• Ensure account meets module pre-requisites from above.

• You may also choose to use this module with a custom AMI image as shown in the existing-image example.

• Please note that while some resources are individually and uniquely tagged, all common tags are expected to be configured within the AWS provider as shown in the example code snippet below.

• Create a Terraform configuration that pulls in this module and specifies values of the required variables:

provider "aws" {
  region = "<your AWS region>"
  default_tags {
    tags = var.common_tags
  }
}

module "tfe_node" {
  source                 = "<filepath to cloned module directory>"
  friendly_name_prefix   = "<prefix for naming AWS resources>"
  domain_name            = "<domain for creating the Terraform Enterprise subdomain on. >"
  tfe_license_secret_id  = data.aws_secretsmanager_secret_version.tfe_license.secret_id
  acm_certificate_arn    = "<ARN for ACM cert to be used with load balancer>"
}

• Run terraform init and terraform apply

Access to the Application Servers

• Cloud-native access to application servers which lie behind load-balancers is recommended over SSH/bastion-based access.
• This module deploys the SSM agent on RHEL (it is already present in the Ubuntu AWS marketplace images), but requires an IAM role policy ARN such as "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" in the iam_role_policy_arns list in order to enable access via SSM. Your options at this time are:
  • Deploy the requisite IAM role policy.
  • Add additional resources to deploy a bastion host as required to be able to access the application hosts on the command line.

Module Manifest

This module will create all infrastructure resources required to install Terraform Enterprise in a standalone or active-active configuration (depending on how many nodes you specify) on AWS in the designated region according to the Reference Architecture. The default base AMI used is Ubuntu 20.04 LTS but you may specify a RHEL 7.x AMI ID by using the ami_id variable.

The resources created are:

• VPC with public and private subnets
• PostgreSQL instance
• Redis cache
• S3 bucket for installation bootstrapping
• Auto-scaling group behind Application Load Balancer (ALB)
• Secrets Manager Secret used for deploys
• KMS key
• IAM Instance Role and IAM Policy to allow instances to retrieve bootstrap secrets
• Route53 A Record for Load Balancer on TFE domain
• Supporting security groups and rules for application functionality

Examples

We have included documentation and reference examples for additional common installation scenarios for TFE, as well as examples for supporting resources that lack official modules.

• Example: Deploying with an existing, custom image
• Example: Deploying with AWS Aurora RDS cluster instance
• Example: Deploying behind a proxy (coming soon...)
• Example: Deploying into an existing private network (coming soon...)
• Example: Deploying while managing DNS outside of AWS (coming soon...)

License

This code is released under the Mozilla Public License 2.0. Please see LICENSE for more details.

terraform-aws-terraform-enterprise's Issues

testing again

What problem would you like to address?

Who is the audience for this request?

What would the ideal solution look like?

What alternatives have you considered?

How does this impact you?

Are there any deadlines?

Is there any additional context?

EC2 Instances are marked as "unhealthy"

We are currently working on the deployment of Terraform Enterprise to our AWS Account and we are facing a problem with the EC2 instances, that we don't really know how to debug (or even find useful logs to investigate further)

Without making any change on the code of the terraform-aws-terraform-enterprise module, all resources are provisioned but as i mentioned previously the ec2 instances (both primary and secondaries) are marked as healthy when they try to register to the TargetGroup.

Health checks failed with these codes: [502]

Any ideas where to find logs to help us investigate this further?

Add ability to pin Terraform Enterprise version

Hello,

This issue is to track adding the ability to pin the Terraform Enterprise version that is installed by this module. In non-clustered installations this was configured using ReleaseSequence within the /etc/replicated.conf file.

Module is specific to v0.11.x will not work for v0.12

I have ran into multiple issues trying to use this module with TF v0.12.13 and I am still working through them.

Can documentation be update to reflect the known working version this module should be used with?

Some key breaking areas are handling variable lists.
Example:

resource "aws_security_group_rule" "allow_list" {
  count             = "${length(var.allow_list) > 0 ? 1 : 0}"
  type              = "ingress"
  protocol          = "-1"
  from_port         = 0
  to_port           = 0
  cidr_blocks       = ["${var.allow_list}"]
  security_group_id = "${aws_security_group.intra_vpc_and_egress.id}"
} 

This breaks because var.allow_list is a list. Had to change to

cidr_blocks  = var.allow_list

To get plan to move past that error. I'm still working through the errors as I need version 0.12 in my current setup.

testing FR

What problem would you like to address?

Who is the audience for this request?

What would the ideal solution look like?

What alternatives have you considered?

How does this impact you?

Are there any deadlines?

Is there any additional context?

Implement check for authorized users of slash command dispatcher

What problem would you like to address?

Currently for someone to run the slash commands (i.e. /test all), the user must be admin of the repo. This security posture is not preferred.

Who is the audience for this request?

TFE engineers / maintainers of this repo

What would the ideal solution look like?

Implement a conditional that is a check of authorized users for the slash command dispatcher action, like done here

What alternatives have you considered?

What we're doing now

How does this impact you?

Currently we have to add everyone as a user and remember to remove admins when they're offboarded.

Are there any deadlines?

Is there any additional context?

Add ability to specify tags for the resources

Hello,

This issue is to track the ability to add tags to the resources managed by this module. Currently, it is required for the user to make manual edits to the module to tag the resources. Perhaps an input variable can be added that a user can pass a map of tags into.

primary_count variable is ignored

The primary count variable is ignored, because the number of primaries is hard-coded to 3.

Testing bug

Expected Behavior

Actual Behavior

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

Create variable for vm health check grace period

What problem would you like to address?

@jkerry created this branch which he's using here.
Who is the audience for this request?

TFE Engineer

What would the ideal solution look like?

The branch is behind main and needs to be updated and merged into main.

airgap example

In the airgapped install example, where is the source for the external module located? Can't seem to locate this source "hashicorp/terraform-enterprise/aws/modules/external-services"

expose load balancer attribute outputs

Can we look at exposing more outputs from the main module such as:

• Load Balancer ARN
• Load Balancer Security Group(s)

Create a bastion vm fixture module and enable ssh to TFE instances in Active/Active mode

What problem would you like to address?

Customers and team members alike have needed an AWS active/active installation that they can easily SSH to, and currently, this module does not support that; instead SSM is strictly enforced.

Who is the audience for this request?

TFE Engineers, internal teams needed test instances

What would the ideal solution look like?

I have started a branch here. You will notice that there is a bastion_host fixture module. I would continue working on this to get it to a streamlined build.

I attempted to build out the solution here, running it in tf-onprem-dev-infra/modules/release-testing, but I didn't have time to complete it.

Ideally, someone would just pick up on this work.

What alternatives have you considered?

Being able to SSH directly to the TFE instances for DEV scenarios, but I'm not sure if that's possible.

How does this impact you?

Not having this adds a lot of toil to building out this scenario manually.

Are there any deadlines?

No, but we waste a lot of time on it.

Is there any additional context?

I had to build a module for someone (convo here) who didn't have access to our AWS account to use SSM, so me and another engineer spent two hours figuring it out manually. Having a prebuilt solution would reduce toil.

Airgap Install Examples make external calls

For v5 architecture, the airgapped install example appears to use 'external' resources like the public module registry (to get modules) and s3 buckets (to get the installer bits). Airgapped implies that the install will not require any sort of external internet access.

Airgapped installer example code:
https://github.com/hashicorp/terraform-aws-terraform-enterprise/blob/master/examples/airgap/main.tf

Calling v5 module from public module registry:
module "terraform-enterprise" { source = "hashicorp/terraform-enterprise/aws" version = "0.1.0" ... }

Calling external s3 bucket:
installer_url = "https://${local.setup_bucket}.s3-${local.region}.amazonaws.com/tfe-setup/ptfe.zip"

Add feature/ability to pass in existing resources instead of assume creation from zero

What problem would you like to address?
Take a govcloud deployment for example. In govcloud, the project may not be able to create VPC or other resources. The govcloud governance and cloud service provider may create those resources for the project. I would like to be able to pass those values in and instead of the module creating the resource, consume the given resource. ie. vpc_id, public_subnets, or IAM_role

Who is the audience for this request?
The end users are initially a govcloud user, but could be any cloud tenant depending on governance.

What would the ideal solution look like?
If the resource has been provided then consume the resource. If the resource is not provided, attempt to create the resources as currently designed to do.

What alternatives have you considered?
A. Writing our own modules.
B. Tearing apart the current upstream deployment to perform resource creation using a different IAM roles.

How does this impact you?
We cannot create iam roles/policies due to govcloud and company governance policies.
We have been working to provide values to each of the services and implement a count in the resources to check if provided.

Are there any deadlines?
1 May 2023

Is there any additional context?
Govcloud account

TEST - AWS

What problem would you like to address?

Who is the audience for this request?

What would the ideal solution look like?

What alternatives have you considered?

How does this impact you?

Are there any deadlines?

Is there any additional context?

Add ability to have the installation done fully on a private subnet

Would be good to give users options as to which subnet they would like TFE to be installed in, rather than assuming everyone wants to use public subnet for primary.

Metrics support (aws)

What problem would you like to address?

Currently there is no configuration for the load balancers or security groups for the metrics ports to be open.

Who is the audience for this request?

all

What would the ideal solution look like?

Here is the Azure solution.
hashicorp/terraform-azurerm-terraform-enterprise#214

What alternatives have you considered?

n/a

How does this impact you?

This is an important customer feature that we need to not manually configuring.

Are there any deadlines?

n/a

Is there any additional context?

update to v5 AWS provider

Expected Behavior

We need this fix for the default tags.

Actual Behavior

If you change default tags, it won't init.

Steps to Reproduce

spin something up on aws provider 4.0
change the default tags
failure

Primary TFE instances do not have access to internet

Hello,
I've noticed that all primary tfe instances do not have access to internet (egress rule is missing).
Without the access some updates are failing. Is this an expected behaviour?

I did enable the access by adding the following egress rule:

resource "aws_security_group" "allow_ptfe" {
...
...
  egress {
    description = "Allow EC2 to have access to internet"

    from_port = 0
    to_port   = 0
    protocol  = "-1"

    cidr_blocks = ["0.0.0.0/0"]
  }
}

Configurable auto scaling group health check

What problem would you like to address?

Support a configurable health check for the auto scaling group. The current settings are too aggressive and cause the instance to terminate during an upgrade.

• Add support for configuring the health_check_grace_period.
• Add support for configuring the health_check_type.

Who is the audience for this request?

Everyone.

What would the ideal solution look like?

Expose variable to configure the aforementioned settings.

What alternatives have you considered?

Using Azure and GCP.

How does this impact you?

It making doing automated upgrade testing difficult.

Are there any deadlines?

N/A

Is there any additional context?

N/A