ulamlabs / terraform-aws-rabbitmq

Terraform configuration for creating RabbitMQ cluster on AWS.

Dead simple Terraform configuration for creating RabbitMQ cluster on AWS.

Branch	Build status
master

What it does ?

1. Uses official RabbitMQ docker image.
2. Creates N nodes in M subnets
3. Creates Autoscaling Group and ELB to load balance nodes
4. Makes sure nodes can talk to each other and create cluster
5. Make sure new nodes always join the cluster
6. Configures / vhost queues in High Available (Mirrored) mode with automatic synchronization ("ha-mode":"all", "ha-sync-mode":"3")

How to use it ?

Copy and paste into your Terraform configuration:

module "rabbitmq" {
  source                            = "ulamlabs/rabbitmq/aws"
  version                           = "3.0.0"
  vpc_id                            = var.vpc_id
  ssh_key_name                      = var.ssh_key_name
  subnet_ids                        = var.subnet_ids
  elb_additional_security_group_ids = var.cluster_security_group_id
  min_size                          = "3"
  max_size                          = "3"
  desired_size                      = "3"
}

then run terraform init, terraform plan and terraform apply.

Are 3 node not enough ? Update sizes to 5 and run terraform apply again, it will update Autoscaling Group and add 2 nodes more. Dead simple.

Node becomes unresponsive ? Autoscaling group and ELB Health Checks will automatically replace it with new one, without data loss.

Note: The VPC must have enableDnsHostnames = true and enableDnsSupport = true for the private DNS names to be resolvable for the nodes to connect to each other.

terraform-aws-rabbitmq's Issues

validation error during apply because of duplicate entries

During apply I get this error:

2 error(s) occurred:

• aws_elb.elb: 1 error(s) occurred:

• aws_elb.elb: ValidationError: Either AvailabilityZones or SubnetIds must be specified
  status code: 400, request id: 550d8253-38dc-11e9-a881-d79abf3541d8

• aws_launch_configuration.rabbitmq: 1 error(s) occurred:

• aws_launch_configuration.rabbitmq: Error creating launch configuration: ValidationError: Must use either use group-id or group-name for all the security groups, not both at the same time
  status code: 400, request id: 5af9c704-38dc-11e9-9116-9de1e4d172ac

Which should be leading and how to fix this?

Thanks,
Sander

Clustering does not work ***NOT WORKING CODE***

Hello,
I have attempted to use your code on AWS. I have been testing this for a week now without any progress. The nodes are configured individually and there is no HA or clustering occurring. The nodes individually have a RabbitMQ broker and the ELB directs traffic in a round robin fashion. The nodes do not mirror and there is no clustering.

For example see below,

oot@ip-10-101-204-179:/# rabbitmqctl cluster_status
Cluster status of node rabbit@ip-10-101-204-179 ...
[{nodes,[{disc,['rabbit@ip-10-101-204-179']}]},
{running_nodes,['rabbit@ip-10-101-204-179']},
{cluster_name,<<"rabbit@ip-10-101-204-179">>},
{partitions,[]},
{alarms,[{'rabbit@ip-10-101-204-179',[]}]}]

Only ONE node is present in each cluster. Could you please fix this issue since there is no HA or clustering currently happening with the code. The problem is with the join_cluster.sh and find_hosts.sh scripts.

Unsupported block type

When running plan on my code, I get these errors. I am running terraform 0.12.6 from terragrunt 0.19.14 on a Mac.

Error: Unsupported block type

  on .terraform/modules/rabbitmq/ulamlabs-terraform-aws-rabbitmq-2889a04/main.tf line 49, in data "template_file" "cloud-init":
  49:   vars {

Blocks of type "vars" are not expected here. Did you mean to define argument
"vars"? If so, use the equals sign to assign it a value.


Error: Unsupported block type

  on .terraform/modules/rabbitmq/ulamlabs-terraform-aws-rabbitmq-2889a04/main.tf line 105, in resource "aws_security_group" "rabbitmq_elb":
 105:   tags {

Blocks of type "tags" are not expected here. Did you mean to define argument
"tags"? If so, use the equals sign to assign it a value.

According to https://www.terraform.io/docs/providers/template/d/file.html and https://www.terraform.io/docs/providers/aws/r/security_group.html those should use an =.

Found this in the upgrade docs (https://www.terraform.io/upgrade-guides/0-12.html):

Due to the design of the configuration language decoder in Terraform v0.11 and earlier, it was in many cases possible to interchange the argument syntax (with =) and the block syntax (with just braces) when dealing with map arguments vs. nested blocks. However, this led to some subtle bugs and limitations, so Terraform v0.12 now requires consistent usage of argument syntax for arguments and nested block syntax for nested blocks.

This leads me to believe that it should be safe to use an equal sign for pre 0.12 versions.

Console is displaying "Management API returned status code 504"

Any idea where I should look. After connecting and logging into the rabbitmq console via the load balancer for 15-30 seconds "Management API returned status code 504 -" displays on the bottom of the screen. I am testing with t2.small. Too small maybe?

Version = "2.0.1".

Clustering does not work - Uppercase Hostnames

Thank you for this script! I has saved us a considerable amount of time. Unfortunately, running this unchanged presented a couple of challenges around clustering. The rabbit nodes were unable to establish an erlang cluster.

I was able to eventually get it to work by adding -e RABBITMQ_USE_LONGNAME=true to the docker run command, and setting to uppercase the hostnames.

I'll submit a pull request for your review around this issue.

Query

Hi; out of curiosity; why do you run rabbit in a docker container within a single instance, within the autoscale group rather than directly on the machine?

instances not passing healthcheck.

hello,

great example, when i run it my instances are not being seen as healthy by the elb?

any ideas?

aws ec2 describe instances NOT RETURNING ANY HOSTS

The command to retrieve the DNSes in the cloud-init.yaml file is not returning any information regarding the private DNSes. Could you please check on this? My cluster has only one node each and the ELB directly routes the messages to them individually without HA.

Name conflict when changing the aws_launch_configuration of a working cluster.

Example: instance type changes.
default (m5.large) -> instance_type = "t2.micro"

-/+ module.rabbitmq.aws_launch_configuration.rabbitmq (new resource required)
      instance_type:                             "m5.large" => "t2.micro" (forces new resource)

....
  Enter a value: yes
module.rabbitmq.aws_launch_configuration.rabbitmq: Creating...
Error: Error applying plan:

1 error(s) occurred:

* module.rabbitmq.aws_launch_configuration.rabbitmq: 1 error(s) occurred:

* aws_launch_configuration.rabbitmq: Error creating launch configuration: AlreadyExists: Launch Configuration by this name already exists - A launch configuration already exists with the name rabbitmq-main

I think it is unnecessary to specify the aws_launch_configuration name if the option "create_before_destroy = true" is used.

sync_node_count

I see that your sync_node_count is hardcoded as 3 in main.tf.

data "template_file" "cloud-init" {
  template = "${file("${path.module}/cloud-init.yaml")}"

  vars {
    sync_node_count = 3
    tcp_tg_arn      = "${aws_lb_target_group.tcp_tg.arn}"
    asg_name        = "${local.cluster_name}"
    region          = "${data.aws_region.current.name}"
    admin_password  = "${random_string.admin_password.result}"
    rabbit_password = "${random_string.rabbit_password.result}"
    secret_cookie   = "${random_string.secret_cookie.result}"
    message_timeout = "${3 * 24 * 60 * 60 * 1000}"              # 3 days
  }
}

Shouldn't this number be dynamic (the number of running instances in the autoscaling group)?

initial ref transaction called with existing refs on init

Running terraform init after providing variables grants the following error

Error downloading modules: Error loading modules: error downloading 'https://github.com/ulamlabs/terraform-aws-rabbitmq?ref=v3.0.0': /usr/bin/git exited with 128: Cloning into '.terraform/modules/092ebc75691ed8de2f41a34a62b78538'...
fatal: BUG: initial ref transaction called with existing refs

Update the EC2 AMIs

I would like to inform you that this module cannot be used anymore due to non-existent EC2 AMIs. The least you can do is update the "aws_ami_ids" with the following filter:
amzn-ami-hvm-2018*-gp2

cloud-init.yaml file is ignored

Today I tried to use this module but despite the correct "deploy" of all AWS resources (Launch Configuration, Auto Scaling Groups, ELB, EC2, Security Groups, IAM role) in EC2 instances, there is no Rabbitmq instance, not even Docker is installed via "yum",

I tried to debug the problem, but I haven't Terraform or EC2 bash errors.

My basic Terraform configuration:

provider "aws" {
  region = "eu-west-1"
}
module "rabbitmq" {
  source        = "ulamlabs/rabbitmq/aws"
  version       = "3.0.0"
  vpc_id        = "vpc-1234"
  ssh_key_name  = "keypair"
  subnet_ids    = ["subnet-123a", "subnet-123b", "subnet-123c"]
  min_size      = "3"
  max_size      = "3"
  desired_size  = "3"
  instance_type = "t2.small"
}

Ephemeral Storage upto 1TiB

Is there a way to setup ephemeral storage in the scripts so that nodes have a higher storage capacity? Our RabbitMQ cluster crashes because of our production workloads. I was wondering if you considered the storage capacity here or could point me in the right direction? Please do the needful.

Clarify license

This project looks great but I'm unfortunately unable to use it without knowing how it is licensed. It would help if that could be clarified :-)

This is a useful resource for making the choice: https://choosealicense.com/

Load Balancer not marking instances as healthy

Script does not work out of the box. Auto scaling group keeps terminating RabbitMQ servers due to load balancer not marking instances as healthy. Since the health check is over port 5672 it suggests that RabbitMQ docker is either not configured right or not starting properly.

Incorrect attribute value type

New to terraform, but seeing the following error messages and not sure what the issue is from following the readme:

Error: Incorrect attribute value type

  on .terraform/modules/rabbitmq/ulamlabs-terraform-aws-rabbitmq-2889a04/main.tf line 156, in resource "aws_launch_configuration" "rabbitmq":
 156:   security_groups      = ["${aws_security_group.rabbitmq_nodes.id}", "${var.nodes_additional_security_group_ids}"]

Inappropriate value for attribute "security_groups": element 1: string
required.


Error: Incorrect attribute value type

  on .terraform/modules/rabbitmq/ulamlabs-terraform-aws-rabbitmq-2889a04/main.tf line 216, in resource "aws_elb" "elb":
 216:   subnets         = ["${var.subnet_ids}"]

Inappropriate value for attribute "subnets": element 0: string required.


Error: Incorrect attribute value type

  on .terraform/modules/rabbitmq/ulamlabs-terraform-aws-rabbitmq-2889a04/main.tf line 219, in resource "aws_elb" "elb":
 219:   security_groups = ["${aws_security_group.rabbitmq_elb.id}", "${var.elb_additional_security_group_ids}"]

Inappropriate value for attribute "security_groups": element 1: string
required.

versions:

source                            = "ulamlabs/rabbitmq/aws"
version                           = "2.0.1"
terraform                         = 0.12.6

legacy elb being used

I noticed this is using the legacy aws_elb instead of aws_lb. Are you willing to accept PR's?
I noticed 2 PR's unmerged from last year that look legit.