ronakongit / hibernate

Rapidly shrinks cluster out of working hours

Hibernate shrinks and expands your cluster on schedule

This utility currently is on best effort support by @Leon Kuperman and @Augustinas Stirbis through community slack

Install hibernate

Run this command to install Hibernate CronJobs

kubectl apply -f https://raw.githubusercontent.com/castai/hibernate/main/deploy.yaml

Change API key

Create API token with Full Access permissions and encode base64

echo -n "98349587234524jh523452435kj2h4k5h2k34j5h2kj34h5k23h5k2345jhk2" | base64

use this value to update Secret

apiVersion: v1
kind: Secret
metadata:
  name: castai-Hibernate
  namespace: castai-agent
type: Opaque
data:
  API_KEY: >-
    CASTAI-API-KEY-REPLACE-ME-WITH-ABOVE==

OR for convenience use one liner

kubectl get secret castai-hibernate -n castai-agent -o json | jq --arg API_KEY "$(echo -n 9834958-CASTAI-API-KEY-REPLACE-ME-5k2345jhk2 | base64)" '.data["API_KEY"]=$API_KEY' | kubectl apply -f -

Set Cloud env variable

AKS is set by default, but requires changing in both CronJobs "Cloud" env variable to [EKS|GKE|AKS]

How it works

Hibernate-pause Job will

• Disable Unscheduled Pod Policy (to prevent growing cluster)
• Prepare Hibernation node (node that will stay hosting essential components)
• Mark essential Deployments with Hibernation toleration (system critical and with NAMESPACES_TO_KEEP env var)
• Delete all other nodes (only hibernation node should stay running)

Hibernate-resume Job will

• Renable Unscheduled Pod Policy to allow cluster to expand to needed size

Override default hibernate-node size

• Set the HIBERNATE_NODE environment variable to override the default node sizing selections. Make sure the size selected is appropriate for your cloud.

Override default NAMESPACES_TO_KEEP

• Set the NAMESPACES_TO_KEEP environment variable to override, "opa,istio""

Override default "PROTECT_EVICTION_DISABLED" and set to "true" to prevent the removal of removal-disabled nodes from being removed during hibernate. This looks for the autoscaling.cast.ai/removal-disabled="true" label on a node and if it exists excludes it from being cordoned and deleted.

TODO

• Auto detect Cloud

Development

Create [aks|eks|gke] K8s cluster

• create file hack/aks/local.auto.tfvars from example
• run "make aks"
• connect to cluster / switch kubectl context

Run code locally

• copy cluster_id from console.cast.ai to .env file (example .env.example)
• uncomment in main.py # local_development = True
• run end2end tests