bayer-group / etcd-aws-cluster

A container to assist in managing a etcd2 cluster from an Amazon auto scaling group

etcd-aws-cluster

This container serves to assist in the creation of an etcd (2.x) cluster from an AWS auto scaling group. It writes a file to /etc/sysconfig/etcd-peers that contains parameters for etcd:

• ETCD_INITIAL_CLUSTER_STATE
  • either new or existing
  • used to specify whether we are creating a new cluster or joining an existing one
• ETCD_NAME
  • the name of the machine joining the etcd cluster
  • this is obtained by getting the instance if from amazon of the host (e.g. i-694fad83)
• ETCD_INITIAL_CLUSTER
  • this is a list of the machines (id and ip) expected to be in the cluster, including the new machine
  • e.g., "i-5fc4c9e1=http://10.0.0.1:2380,i-694fad83=http://10.0.0.2:2380"

This file can then be loaded as an EnvironmentFile in an etcd2 drop-in to properly configure etcd2:

[Service]
EnvironmentFile=/etc/sysconfig/etcd-peers

Workflow

• get the instance id and ip from amazon

• fetch the autoscaling group this machine belongs to

• obtain the ip of every member of the auto scaling group

• for each member of the autoscaling group detect if they are running etcd and if so who they see as members of the cluster

  if no machines respond OR there are existing peers but my instance id is listed as a member of the cluster

  • assume that this is a new cluster
  • write a file using the ids/ips of the autoscaling group

  else

  • assume that we are joining an existing cluster
  • check to see if any machines are listed as being part of the cluster but are not part of the autoscaling group
    • if so remove it from the etcd cluster
  • add this machine to the current cluster
  • write a file using the ids/ips obtained from query etcd for members of the cluster

Usage

docker run -v /etc/sysconfig/:/etc/sysconfig/ monsantoco/etcd-aws-cluster

Environment Variables

• PROXY_ASG - If specified forces into proxy=on and uses the vaue of PROXY_ASG as the autocaling group that contains the master servers
• ASG_BY_TAG - If specified in conjunction with PROXY_ASG uses the value of PROXY_ASG to look up the server by the Tag Name
• ETCD_CLIENT_SCHEME - defaults to http
• ETCD_PEER_SCHEME - defaults to http

Demo

We have created a CloudFomation script that shows sample usage of this container for creating a simple etcd cluster: https://gist.github.com/tj-corrigan/3baf86051471062b2fb7

etcd-aws-cluster's Issues

Update to use 3.6

I started to test using the tool to bootstrap against etcd3 cluster with v2 mode. Noticed the alpine version is still 3.4. Would you consider to upgrade to gliderlabs/alpine:3.6?

Port settings broken because of merge conflict

The client/server port settings introduced in f3b55e5 were subsequently clobbered by ea5bb27

See line 57 and line 156 in the latter change. Instead of referencing bash variables, the use hard-coded port numbers.

Bug - Problem with spaces in autoscaling group

Hey Ho,

first - Thanks for your awesome script. It works like a charm :)
Just found a small bug regarding Autoscaling Group Names with spaces in the name.
Its fixed if you use quotes around the variable.
I will create a Pull Request for it. It would be great if you could merge it afterwards.

Greetings, Thomas

Edit: Added reference to Pull Request #19

Having issues with autoscaling.

Issue I'm running into:

Steps:
Start a new cluster with 3 nodes... Etcd comes up okay.

Autoscale to 6 nodes..

Etcd fails on new nodes:
Jun 03 18:38:03 ip-10-7-2-18.ec2.internal docker[5987]: 2016/06/3 18:38:03 etcdmain: error validating peerURLs {ClusterID:8a188635e6676cf7 Members:[&{ID:5db898d9b2673eb3 RaftAttributes:{PeerURLs:[http://10.7.2.18:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:610c3a0ab98d72b7 RaftAttributes:{PeerURLs:[http://10.7.1.167:2380]} Attributes:{Name:i-fcdc0904 ClientURLs:[http://10.7.1.167:2379]}} &{ID:80c9803aa2fa7908 RaftAttributes:{PeerURLs:[http://10.7.2.17:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:e23dd231e95a85b4 RaftAttributes:{PeerURLs:[http://10.7.1.205:2380]} Attributes:{Name: ClientURLs:[]}} &{ID:e8071ff486efa6f2 RaftAttributes:{PeerURLs:[http://10.7.1.166:2380]} Attributes:{Name:i-fbdc0903 ClientURLs:[http://10.7.1.166:2379]}} &{ID:fe69d87bb0fad3a0 RaftAttributes:{PeerURLs:[http://10.7.2.16:2380]} Attributes:{Name:i-96589a10 ClientURLs:[http://10.7.2.16:2379]}}] RemovedMemberIDs:[]}: member count is unequal

Here is the generated file:
cat /etc/sysconfig/etcd-peers

ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_NAME=i-32ac62b4
ETCD_INITIAL_CLUSTER="i-fcdc0904=http://10.7.1.167:2380,=http://10.7.1.205:2380,i-fbdc0903=http://10.7.1.166:2380,i-96589a10=http://10.7.2.16:2380,i-32ac62b4=http://10.7.2.18:2380"

I suspect it's the missing ID above in the initial cluster list which is breaking things.

Any help would be appreciated!

DNS Names as endpoints

Is it possible to use DNS names rather that IP addresses?

Docker image not up to date

Seems like there is a build trigger problem (or a single missed build) as monsantoco/etcd-aws-cluster:latest does not contain the fix for setting custom client/server ports that is in the code in repo

Bootstrap script should handle clusters set up to use client certs

This line will not work if the etcd cluster is configured to use SSL client certificates.

There should be a way to specify --key, --cert, and --cacert parameters to that curl command to make sure that curl can communicate with any existing peers.

curl -f -s http://ASG_SERVER_IP:2379/v2/members hangs

Hello Monsanto,
Great process and flow. I was able to bootstrap a cluster pretty quickly.

I've read ETCd clustering post a couple of times and couldn't find a solution/reason to the following:
When the ASG bootstraps the ETCd instances, sporadically, the etc-aws-cluster script would hang when executing the /members cURL command: curl -f -s http://ASG_SERVER_IP:2379/v2/members.

Lets take hosts A and B.
Host A is running etcd-aws-cluster script as part of the cloud-init phase. tried to execute the /members API call on host B. the cURL command hangs.
Only when I restart the ETCd systemd.unit on host B, host A can continue the script and the cluster gets bootstrapped.

Hope the description is clear enough for reproduce.

I'm available if more info is required.

Thank you in advance,
Yarden

Licensing Question

So, just a quick question on the licensing terms of this project.

I'm currently running a regular etcd cluster inside of a VPC on AWS. I'd like to spin up a new VPC for a trial run of etcd-aws-cluster.

The issue is that I'd like to link the two VPCs for centralized log management / monitoring / etc. I'm concerned that this transfer of packets may constitute a kind of digital "cross pollination" which could result in my losing all legal rights to the existing cluster.

Is this something I should be concerned about?
Please advise.

etcd_initial_cluster question

@tj-corrigan This is awesome! 👍

Just one question, what is the difference between creating "etcd_initial_cluster" when joining or creating a brand new cluster? Does it matter if we completely ignore what etcd is telling us and take into account just what autoscaling group have in its current instance ip list?

Something like:

[...]


etcd_initial_cluster=$(aws ec2 describe-instances --region $region --instance-ids $(aws autoscaling describe-auto-scaling-groups --region $region --auto-scaling-group-name $asg_name | jq .AutoScalingGroups[0].Instances[].InstanceId | xargs) | jq -r '.Reservations[].Instances | map(.InstanceId + "=http://" + .NetworkInterfaces[].PrivateIpAddress + ":2380")[]' | xargs | sed 's/  */,/g')
echo "etcd_initial_cluster=$etcd_initial_cluster"
if [[ ! $etcd_initial_cluster ]]; then
    echo "$pkg: unable to get peers from auto scaling group"
    exit 10
fi

cat > "$etcd_peers_file_path" <<EOF
ETCD_NAME=$ec2_instance_id
ETCD_INITIAL_CLUSTER="$etcd_initial_cluster"
EOF

# if I am already listed as a member of the cluster assume that this is a new cluster
if [[ $etcd_existing_peer_urls && $etcd_existing_peer_names != *"$ec2_instance_id"* ]]; then
    echo "joining existing cluster"

    # eject bad members from cluster
    peer_regexp=$(echo "$etcd_peer_urls" | sed 's/^.*http:\/\/\([0-9.]*\):[0-9]*.*$/contains(\\"\1\\")/' | xargs | sed 's/  */ or /g')
    if [[ ! $peer_regexp ]]; then
        echo "$pkg: failed to create peer regular expression"
        exit 6
    fi

    echo "peer_regexp=$peer_regexp"
    bad_peer=$(echo "$etcd_members" | jq --raw-output ".[] | map(select(.peerURLs[] | $peer_regexp | not )) | .[].id")
    echo "bad_peer=$bad_peer"

    if [[ $bad_peer ]]; then
        for bp in $bad_peer; do
            echo "removing bad peer $bp"
            status=$(curl -f -s -w %{http_code} "$etcd_good_member_url/v2/members/$bp" -XDELETE)
            if [[ $status != $delete_ok ]]; then
                echo "$pkg: ERROR: failed to remove bad peer: $bad_peer, return code $status."
                exit 7
            fi
        done
    fi

    # join an existing cluster
    echo "adding instance ID $ec2_instance_id with IP $ec2_instance_ip"
    status=$(curl -f -s -w %{http_code} -o /dev/null -XPOST "$etcd_good_member_url/v2/members" -H "Content-Type: application/json" -d "{\"peerURLs\": [\"http://$ec2_instance_ip:2380\"], \"name\": \"$ec2_instance_id\"}")
    if [[ $status != $add_ok && $status != $already_added ]]; then
        echo "$pkg: unable to add $ec2_instance_ip to the cluster: return code $status."
        exit 9
    fi

    echo "ETCD_INITIAL_CLUSTER_STATE=existing" >> "$etcd_peers_file_path"
else
    # create a new cluster
    echo "creating new cluster"
    echo "ETCD_INITIAL_CLUSTER_STATE=new" >> "$etcd_peers_file_path"
fi
[...]