julie-ng / cloudkube-aks-clusters

3 Clusters, 1 Repo. Opinionated infrastructure as code for my Azure Kubernetes clusters for running demo apps.

Hallo! Follow Me on YouTube to learn more about cloud engineering best practices and watch me code 👩‍💻 🎥

cloudkube-aks-clusters's Issues

Add terraform-docs to autogenerate documentation for tf modules

Try auto-generating documentation for my aks-cluster module

using
https://github.com/terraform-docs/terraform-docs/

Move User Assigned MIs outside of Node RG?

Right now if the cluster needs to be rebuilt (Terraform likes that) all the Ingress MI Role assignments to get TLS certificates need to be reset because the identities die with the node resource group.

Moving them outside of Node RG requires a bit more work
https://azure.github.io/aad-pod-identity/docs/getting-started/role-assignment/#user-assigned-identities-that-are-not-within-the-node-resource-group

Try Workload Identity

Currently in Preview
https://docs.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation

Reboot nodes on cluster version upgrade

Version Upgrades require node reboots

In the terraform code, we'll just listen to the orchestrator_version (k8s version on nodes) and run the upgrade command when that changes.

We will not use kubernetes_version, which applies to the control plane and should be upgraded first before the node pools.

Background

AKS does not automatically reboot nodes to actually use newer images - to avoid potentially breaking user workloads.

Per Apply security and kernel updates to Linux nodes in Azure Kubernetes Service (AKS)

To protect your clusters, security updates are automatically applied to Linux nodes in AKS. These updates include OS security fixes or kernel updates. Some of these updates require a node reboot to complete the process. AKS doesn't automatically reboot these Linux nodes to complete the update process.

and later…

Some security updates, such as kernel updates, require a node reboot to finalize the process. A Linux node that requires a reboot creates a file named /var/run/reboot-required. This reboot process doesn't happen automatically.

Additionally updates apply to existing nodes. But new nodes will run the old image

Unattended upgrades apply updates to the Linux node OS, but the image used to create nodes for your cluster remains unchanged. If a new Linux node is added to your cluster, the original image is used to create the node.

Best practice - managed upgrades

For for zero downtime upgrades, run az aks upgrade… which will

• Create new nodes with latest updates
• AKS makes sure no new workloads land that, reserving it for the existing workloads (cordoning)
• Move existing workloads to this new node (draining)
• When finished
  • Restart old nodes
  • New nodes can take new workloads

Details described in this doc

Use Azure `mode` to separate user & system node pools

Problem

Currently the default_node_pool block is used for the system node pool. But this defaults to user mode.

Consequence

AKS has a mode property that can be either system or user which beyond semantics adds the CriticalAddonsOnly=true:NoSchedule taint.

Docs Referender: Manage system node pools in Azure Kubernetes Service (AKS) > System and user node pools

Changes required

• Swap default_node_pool block and the kubernetes_cluster_node_pool resource definitions.
• Will probably destroy and re-create the cluster.

Remove Azure Pod Identity

Reason

Per Azure Pod Identity project repo, Pod Identity will be replaced by workload identity federation (preview) in the future.

❗ IMPORTANT: As mentioned in the announcement, we are planning to replace AAD Pod Identity with Azure Workload Identity. Going forward, we will no longer add new features to this project in favor of Azure Workload Identity. However, we will continue patching critical bugs and security vulnerabilities until further notice.

While Workload Identities are in preview, I won't try that in this change yet.

New Identity/Credential Options

So taking pod identity off the table, we have:

• Workload Based Access Control needed, e.g. multi-tenant security boundary? Use Service Principals
• Single-tenant security boundary? I'm going to use the Kubelet identity on the nodes that's actually shared by all workloads.

Code Changes

• Rip out the pod identity workload and all the extra YAML that the Ingress controller needs just for TLS secret.
• Upgrade to latest Azure Key Vault CSI v1.1.0 release from 2 March - preparing to test workload identity. Managed add-ons are often not latest version.

Create own Managed Identity for Kubelet

Currently code relies on ARM to create the identity. Instead specify own via https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#kubelet_identity