nanit / kubernetes-rabbitmq-cluster

Deploy-ready rabbitmq cluster for kubernetes

kubernetes-rabbitmq-cluster

A ready to deploy rabbitmq cluster to work on top of Kubernetes.

It uses rabbitmq clusterer plugin to achieve clustering.

Requirements:

1. Kubernetes version 1.5.X (We're using StatefulSet)
2. kubectl configured to work with your Kubernetes API
3. Tested on Kubernetes 1.5.2 on top of AWS (See future work)
4. Optional - Access to your own docker repository to store your own images. That's relevant if you don't want to use the default images offered here.

Contents:

1. A 3 nodes rabbitmq cluster as StatefulSet
2. A rabbitmq-headless headless service to control the StatefulSet domain
3. a rabbitmq service to access the cluster
4. An optional, rabbitmq-management service to access the admin control panel

Environment Variables:

Name	Default Value	Purpose
NAMESPACE	default	Change it if you want to create the RabbitMQ cluster in a custom Kubernetes namespace. If the namespace does not exist in the moment of deployment, it will be created for you.
DOCKER_REPOSITORY	nanit	Change it if you want to build and use custom docker repository
POD_MEMORY_REQUEST	None	memory allocated per pod by default
POD_CPU_REQUEST	None	cpu per pod
SUDO	sudo	Should docker commands be prefixed with sudo. Change to "" to omit sudo.
RBAC	FALSE	Should create a role/system account and role binding
RABBITMQ_REPLICAS	3	Number of nodes in the cluster
RABBITMQ_DEFAULT_USER	None	The default username to access the management console
RABBITMQ_DEFAULT_PASS	None	The default password to access the management console
RABBITMQ_ERLANG_COOKIE	None	Erlang secret needed for nodes communication
RABBITMQ_EXPOSE_MANAGEMENT	FALSE	Should RMQ management console be exposed as a service
RABBITMQ_MANAGEMENT_SERVICE_TYPE	LoadBalancer	Kubernetes service type for the management console
RABBITMQ_HA_POLICY	None	Set this variable to automatically set HA policy on all queues
RABBITMQ_LOG_LEVEL	info	Log levels are set for all RabbitMQ log types: connection, mirroring, channel and federation. Valid values are: none, error, warning, info, debug
RABBITMQ_ADDITIONAL_YAML	""	Inject additional arbitrary YAML into the stateful set

Deployment:

1. Clone this repository
2. Run:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export POD_CPU_REQUEST=300m && \
export POD_MEMORY_REQUEST=5Gi && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export RABBITMQ_LOG_LEVEL=info && \
export RABBITMQ_ADDITIONAL_YAML="" && \
export SUDO="" && \
make deploy

Usage:

At the end of the installation you should have a service named rabbitmq which you can use to connect to the cluster. If you've set the environment variable RABBITMQ_HA_POLICY a policy named ha-all is created to match all queues. These are the resource you're supposed to see on the cluster when running kubectl get pods,svc:

Please note the following:

1. The number of rabbitmq pods may be different in case you chose a different number of RABBITMQ_REPLICAS than 5
2. The rabbitmq-management service will only be available if you've set RABBITMQ_EXPOSE_MANAGEMENT to TRUE

Changing the number of nodes:

Number of nodes is configurable with RABBITMQ_REPLICAS environment variable. Note that changing the number of nodes after the initial deployment of the cluster is problematic since old rabbitmq instance won't fetch the new nodes hosts into the clusterer.config file.

For now, the best option is to:

1. Delete the current statefulset with kubectl delete statefulset rabbitmq
2. Re-deploy the cluster with the new RABBITMQ_REPLICAS value

Building your own images:

If you want to build use your own images make sure to change the DOCKER_REPOSITORY environment variable to your own docker repository. It will build the images, push them to your docker repository and use them to create all the needed kubernetes deployments.

Docker Compose:

You can run the same setup in docker-compose using

$ docker-compose build && docker-compose up

Then, go to localhost:15672 and you'll see the cluster is already formed up.

kubernetes-rabbitmq-cluster's Issues

Setting High Availability Policy (HA) is not consistent

Sometimes the pod is ready but still the exec command fails
Should probably use rabbitmqctl wait

Currently, running the deployment command again solves the problem

Kubernetes v1.8 - rabbitmq.config permission denied

Hello,

When I deployed the rabbitmq in kubernetes v1.8, rabbitmq-0 pod returned this error:

/etc/rabbitmq/rabbitmq.config: Permision denied

The same configuration worked well in kubernetes v1.7.

Potential fix is to update the Dockerfile to add the following line after add of clusterer.config:

RUN chmod 666 /etc/rabbitmq/*

Fails on Google Container Engine with namespace other than 'default'

If you change the value of export NAMESPACE=default to something other than default this curl command fails and the whole deployment fails. After I deleted everything and ran it again with namespace set to default it all worked fine.

kubernetes-rabbitmq-cluster/docker/set_cluster_nodes.sh

Line 8 in 045e2e1

STATEFUL_SETS=$(curl -f -k https://kubernetes/apis/apps/v1beta1/statefulsets -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

Perhaps you might have to specify a hostname that includes the namespace?

https://stackoverflow.com/questions/30690186/how-do-i-access-the-kubernetes-api-from-within-a-pod-container#comment75562617_33109739

Waiting for RabbitMQ pod to be ready....

Error:

+ echo 'Waiting for RabbitMQ pod to be ready....'
Waiting for RabbitMQ pod to be ready....
++ kubectl get pods -n test
++ grep rabbitmq-0
++ grep Running
+ [[ -n '' ]]
+ echo 'RabbitMQ pod still not ready...'
RabbitMQ pod still not ready...
+ sleep 5

Steps to Reproduce

1. git clone
2. export the required environment variables
3. make deploy.

The above error just keeps repeating for indefinite period. All the K8s pods seem to be working fine though. Any help would be deeply appreciated.

How do you solve this situation? Why does this happen

[root@node01 kubernetes-rabbitmq-cluster]# kubectl get pod |grep rabbit
rabbitmq-0 1/1 Running 0 18m
rabbitmq-1 0/1 CrashLoopBackOff 8 18m
rabbitmq-2 1/1 Running 2 18m

[root@node01 kubernetes-rabbitmq-cluster]# kubectl get svc |grep rab
rabbitmq ClusterIP 10.111.1.199 5672/TCP,4369/TCP,25672/TCP 19m
rabbitmq-management LoadBalancer 10.108.55.45 15672:31449/TCP 19m

[root@node01 kubernetes-rabbitmq-cluster]# kubectl logs rabbitmq-1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16652 0 16652 0 0 246k 0 --:--:-- --:--:-- --:--:-- 246k
ERROR: epmd error for host rabbitmq-1.rmq-cluster: nxdomain (non-existing domain)

[root@node01 kubernetes-rabbitmq-cluster]# kubectl exec rabbitmq-0 -- rabbitmqctl cluster_status
Cluster status of node '[email protected]' ...
[{nodes,[{disc,['[email protected]']}]},
{running_nodes,['[email protected]']},
{cluster_name,<<"[email protected]">>},
{partitions,[]},
{alarms,[{'[email protected]',[]}]}]

Kubernetes 1.8.2 curl: (22) The requested URL returned error: 403 Forbidden

Hello,

after upgrading Kubernetes cluster to 1.8.2 pods are crashing. I can see next error in logs:
curl: (22) The requested URL returned error: 403 Forbidden

I suspect the problem is in the next string of set_cluster_nodes.sh script:

STATEFUL_SETS=$(curl -f -k https://${KUBERNETES_SERVICE_HOST}/apis/apps/v1beta1/statefulsets -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)")

For some reason the url is not accessible.

Allow setting log levels via environment variable

Incorrect ENV VAR in example run command in README

The example run command says to run:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_EXPOSE_MANAGEMENT=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export SUDO="" && \
make deploy

This is broken, since it repeats the RABBITMQ_EXPOSE_MANAGEMENT twice, with the second time overwriting TRUE with LoadBalancer which is incorrect. The proper var to use is RABBITMQ_MANAGEMENT_SERVICE_TYPE

The proper sample should be:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export SUDO="" && \
make deploy

Allow dynamic number of RabbitMQ nodes

i can't make

i running this turn.

Deployment:

1. Clone this repository
2. Run:

export NAMESPACE=default && \
export DOCKER_REPOSITORY=nanit && \
export RABBITMQ_REPLICAS=5 && \
export RABBITMQ_DEFAULT_USER=username && \
export RABBITMQ_DEFAULT_PASS=password && \
export RABBITMQ_ERLANG_COOKIE=secret && \
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE && \
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer && \
export RABBITMQ_HA_POLICY='{\"ha-mode\":\"all\"}' && \
export RABBITMQ_LOG_LEVEL=info && \
export RABBITMQ_ADDITIONAL_YAML="" && \
export SUDO="" && \
make deploy

but. i'm not success.
i don't know.  why i have this error?  
how i do i it? 

[root@hyperm-1 kubernetes-rabbitmq-cluster-master]# make deploy
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
fatal: Not a git repository (or any of the parent directories): .git
sed -i -e 's/{{RABBITMQ_HA_POLICY}}/{\"ha-mode\":\"all\"}/g' docker/set_ha.sh
docker pull nanit/rabbitmq: || ( docker build -t nanit/rabbitmq: docker &&  docker push nanit/rabbitmq:)
invalid reference format
invalid argument "nanit/rabbitmq:" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
make: *** [docker-rabbitmq] Error 125

Having Issues on k'8 1.16 cluster.

Upgraded cluster from 1.15 to 1.16 pods are failing with exit code 22 error.

Please provide guidance on this issue?

RBAC support for 1.6.x clusters

in rabbitmq-0 logs getting error
curl: (22) The requested URL returned error: 403 Forbidden
would you have an idea of the cause

Add envvars for expiry and length policy

This policy may not be desirable for everyone, and is currently being reapplied every time the deployment restarts.

kubernetes-rabbitmq-cluster/docker/set_ha.sh

Line 11 in fc0f1e8

rabbitmqctl set_policy expiry-and-length '.*' '{"expires":1800000, "max-length":200000}' --apply-to queues || break

Can't deploy rabbitmq in different namespace

Hi i have enabled two contexts in kube prod and dev. In prod i have installed and properly working rabbitmq.

Now i trying to deploy rabbitmq in clear dev namespace, but without success.

My steps:

export NAMESPACE=dev &&
export DOCKER_REPOSITORY=nanit &&
export RABBITMQ_REPLICAS=3 &&
export RABBITMQ_DEFAULT_USER=username &&
export RABBITMQ_DEFAULT_PASS=password &&
export RABBITMQ_ERLANG_COOKIE=devrabbit &&
export RABBITMQ_EXPOSE_MANAGEMENT=TRUE &&
export RABBITMQ_MANAGEMENT_SERVICE_TYPE=LoadBalancer &&
export RABBITMQ_HA_POLICY='{"ha-mode":"all"}' &&
export RABBITMQ_LOG_LEVEL=info &&
export RABBITMQ_ADDITIONAL_YAML="" &&
export SUDO="" &&
make deploy

.....
rabbitmq-0 pod ready ,setting ha policy: {"ha-mode":"all"}

• sleep 20
• kubectl exec -n dev rabbitmq-0 -- rabbitmqctl set_policy ha-all '.*' '{"ha-mode":"all"}' --apply-to queues
  error: unable to upgrade connection: container not found ("rabbitmq")
  Makefile:55: recipe for target 'deploy-rabbitmq' failed
  make: *** [deploy-rabbitmq] Error 1

all 3 replicas in Error state:

NAME READY STATUS RESTARTS AGE
rabbitmq-0 0/1 Error 1 14s
rabbitmq-1 0/1 Error 1 12s
rabbitmq-2 0/1 Error 1 11s

$kubectl logs rabbitmq-0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 47828 0 47828 0 0 834k 0 --:--:-- --:--:-- --:--:-- 819k
/set_cluster_nodes.sh: line 12: ((: 5
3: syntax error in expression (error token is "3")

all free pods - same error:

/set_cluster_nodes.sh: line 12: ((: 5
3: syntax error in expression (error token is "3")

deploy-rabbitmq error 1

hello
so following this per the instructions , after i run make deploy it craps out with the error logs listed below. i think it does not detect all the exports
Some hints.. ? thanks

make deploy
docker pull nanit/rabbitmq:133aedd || ( docker build -t nanit/rabbitmq:133aedd docker && docker push nanit/rabbitmq:133aedd)
Trying to pull repository docker.io/nanit/rabbitmq ...
133aedd: Pulling from docker.io/nanit/rabbitmq
Digest: sha256:392c129dee79739488e15cdfc195d9c2d0babb8aabcab186433f6b546428b5e8
kubectl get ns default || kubectl create ns default
NAME STATUS AGE
default Active 11d
kubectl get svc -n default rabbitmq || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rabbitmq/g' kube/svc.yml | kubectl create -n default -f -
Error from server (NotFound): services "rabbitmq" not found
service "rabbitmq" created
kubectl get svc -n default rmq-cluster || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rmq-cluster/g' kube/svc.headless.yml | kubectl create -n default -f -
Error from server (NotFound): services "rmq-cluster" not found
service "rmq-cluster" created
if [ "TRUE" = "TRUE" ]; then kubectl get svc -n default rabbitmq-management || sed -e 's/{{APP_NAME}}/rabbitmq/g;s/{{SVC_NAME}}/rabbitmq-management/g' kube/svc.management.yml | kubectl create -n default -f - ; fi
Error from server (NotFound): services "rabbitmq-management" not found
service "rabbitmq-management" created
if [ -z "5" ]; then echo "ERROR: RABBITMQ_REPLICAS is empty!"; exit 1; fi
if [ -z "dimtheo" ]; then echo "ERROR: RABBITMQ_DEFAULT_USER is empty!"; exit 1; fi
if [ -z "123456" ]; then echo "ERROR: RABBITMQ_DEFAULT_PASS is empty!"; exit 1; fi
if [ -z "secret" ]; then echo "ERROR: RABBITMQ_ERLANG_COOKIE is empty!"; exit 1; fi
sed -e 's/{{SVC_NAME}}/rmq-cluster/g;s/{{APP_NAME}}/rabbitmq/g;s,{{IMAGE_NAME}},nanit/rabbitmq:133aedd,g;s/{{REPLICAS}}/5/g;s/{{RABBITMQ_DEFAULT_USER}}/dimtheo/g;s/{{RABBITMQ_DEFAULT_PASS}}/123456/g;s/{{RABBITMQ_ERLANG_COOKIE}}/secret/g' kube/stateful.set.yml | kubectl apply -n default -f -
error: unable to decode "STDIN": [pos 305]: json: expect char '"' but got char '1'
make: *** [deploy-rabbitmq] Error 1

How to enable TLS support?

Hello,
could you please advise how to enable TLS support? I'm trying to define environment variables in statefulset manifest:

• name: RABBITMQ_SSL_CERTFILE
  value: /etc/ssl/certs/server.pem
• name: RABBITMQ_SSL_KEYFILE
  value: /etc/ssl/certs/server.key
• name: RABBITMQ_SSL_CACERTFILE
  value: /etc/ssl/certs/ca.pem

and according to docker-endpoint.sh it should work, but it does not.
Thank you.

Huge amount of worker.heartbeat

Hi, I have a huge amount of worker.heartbeat messages in my cluster (around 6 millions), which seems to came progressively since a few hours (cluster works from ~10 days).

Do you know where it could come from ?

Exchange	celeryev
Routing Key	worker.heartbeat
Redelivered	●
Properties	
priority:	0
delivery_mode:	2
headers:	
hostname:	[email protected]
content_encoding:	utf-8
content_type:	application/json
Payload
309 bytes
Encoding: string
{"sw_sys": "Linux", "clock": 9033333, "timestamp": 1498082536.210317, "hostname": "[email protected]", "pid": 34, "sw_ver": "3.1.18", "utcoffset": -2, "loadavg": [1.26, 1.3, 1.47], "processed": 0, "active": 0, "freq": 2.0, "type": "worker-heartbeat", "sw_ident": "py-celery"}

rabbitmq service as a LoadBalancer type

Hey,

Is there a to create rabbitmq service as a LoadBalancer type (in case the 'ha' policy is set to all).
In addition, how are we suppose to access rabbitmq service from the outside, if its type is 'CLUSTER_IP' type?

Thanks for any help

Data persistence

Does this have data persistence? I want to create a new rabbitqm user and it has to survive pod termination. I don't see any persistent volume claims in the kubernetes yaml files. Please correct me if I'm wrong. Thank you.

Handle network partitions

Cant create cluster on k8s

Hi there, when i try to apply the statefulset, my pods fire up but error out after a few seconds.

When i check the k8s logs on the pods, I get the following:

/set_cluster_nodes.sh: line 20: NODES[@]: unbound variable

Can you maybe point me in any direction to fix this?

TIA