Cant emulate node draining within 2 minutes about amazon-ec2-metadata-mock

[bwagner5]

You can setup a Pod Disruption Budget (PDB) on your pods which will ensure that a configured number are always running. If a spot interruption notice is received, aws-node-termination-handler will attempt to cordon and drain the node but the PDB could stall the evictions if evicting would cause the pod count to drop below your configured threshold. But keep in mind that if the 2-minute window ends, your pods may be forced to be shutdown ungracefully since the instance itself will be automatically shutdown.

You can read more about Pod Disruption Budgets here: https://kubernetes.io/docs/concepts/workloads/pods/disruptions/

[roeP]

Thank you for the explanation, I just set the PDB and indeed its not allowing the eviction on the pods.
However the cluster-autoscaler doesn't receive an event to scale the nodes and move the pods to the new one. I know this is question not related to here but since we are talking what would be the best approach to handle that?

[bwagner5]

@roeP the PDB is only going to prevent the eviction, you would need another mechanism to trigger a scale up since cluster-autoscaler will only scale up based on the "pending pods" metric. You can over-provision your cluster to achieve no downtime when a node disappears. That is probably your best bet right now. Over provisioning involves scheduling a configurable amount of headroom via low priority pause-pods. This will ensure that you have space available for pods on another node.

When you receive a spot interruption notice, the aws-node-termination-handler will try to evict and then your pods will get rescheduled on-top of the pause-pods that were over provisioned for you.

NOTE: the following functionality does not exist today and probably doesn't make sense to add:
The aws-node-termination-handler could have functionality added to look at the pods it is evicting and scale up the number of pods on the node that is being drained after it cordons the node. I'm not sure this would be very helpful though. If a node has not already been booted, it's unlikely to have a node be launched and the pods rescheduled within the 2 minute window, so the better solution is still over-provisioning the cluster.

Hope that helps

More info on cluster-overprovisioner:

• https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#how-can-i-configure-overprovisioning-with-cluster-autoscaler
• https://medium.com/scout24-engineering/cluster-overprovisiong-in-kubernetes-79433cb3ed0e

[roeP]

Thank you. That was helpful. Will try it