kwohlfahrt / etcdmon

A script to monitor stacked k8s etcd instances, and remove them on node termination.

etcdmon

This tool helps administer kubernetes control-planes using the stacked etcd topology, by deleting etcd members when their kubernetes node is removed.

If these members are not deleted, and a new node is brought up to replace it, kubeadm join on the new node will fail, when it detects an unhealthy etcd node. By automatically removing the etcd member, the control-plane can be backed by e.g. an AWS autoscaling group, that automatically replaces failed nodes.

Alternatives

An alternatve approach is to use persistent storage for the nodes, as described in this post by Monzo. This requires one auto-scaling group per control-plane node, which makes it difficult to automate rolling upgrades of the nodes.

etcdmon's Issues

Occasional missed node deletion

Occasionally, a node will be deleted and not be detected by etcdmon. This seems to be particularly likely when etcdmon is scheduled on the terminated node during an instance refresh.