learnk8s / kubernetes-production-best-practices

A checklist of Kubernetes best practices to help you release to production

Kubernetes production best-practices

This checklist provides actionable best practices for deploying secure, scalable, and resilient services on Kubernetes.

Topics

• Application development
• Governance
• Cluster configuration

kubernetes-production-best-practices's Issues

More friendly format

@kaihendry suggested a different format: https://webconverger.com/faq/

Perhaps we could think about including a table of content.

use kube2iam in favor of disabling metadata endpoints

This checklist:

Disable metadata cloud providers metada[sic] API

In AWS, this advice precludes containerized applications from using for example AWS IAM role based authentication, a security best practice: credentials are not hard coded, are automatically rotated, and will likely never leave the cluster itself. For more information on this best practice, read here: https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html#use-roles

As a best practice to use this, I would recommend using kube2iam (https://github.com/jtblin/kube2iam) -- this allows you to control which roles a pod is allowed to assume based on namespace, but also allows the pod to see itself as having the role vs requiring it to assume such a role.

In our setup we use kube2iam and assign nodes their IAM role - this IAM role is allowed to assume other roles. With kube2iam, we assign the role through a pod annotation, so the pod sees itself as having that role. Additionally we use namespace restrictions to control access to which roles a pod is allowed to assume.

which is the repo's license?

thanks for this awesome resource!

the text on learnk8s.io says the repo is open source - could you perhaps add a license?

Checklist v2

Feature ideas for v2 of the checklist

➡️ Contributions are welcome! Just add your ideas to this issue. ⬅️

1. Uniform structure for all items
   • Proposal: What?, Why?, How?, References
   • Makes it easier to comprehend items (and easier to write them)
2. Organise items by tags
   • No hardcoded categories. Makes it easier to discover items, but also to add items: no need to decide which category to add an item to, just add the item and assign it a set of relevant tags.
3. Incremental search
   • Incremental search that displays matching items as the search term is typed.
   • Search based on title, content, and possibly tags (if the tag-based organisation is implemented)
4. Tag selection
   • Display all available tags and allow to filter out the items matching one or more selected tag (if the tag-based organisation is implemented)
5. Table of contents
   • If not using a tab-based system, include a table of contents with all items
   • Example: https://webconverger.com/faq/
   • Could also be used with a tag-based system to give a concise overview of the currently selected items

Improve the "Resources utilisation" section

Please review my blog about Kubernetes Resources Management:

• Intro to Kubernetes resources
• What are CPU requests
• The reason for removing CPU Limits

I think you should adjust the "Resources utilization" section.

Include and review kube-score

From kube-score readme:

• Container limits (should be set)
• Pod is targeted by a NetworkPolicy, both egress and ingress rules are recommended
• Deployments and StatefulSets should have a PodDisruptionPolicy
• Deployments and StatefulSets should have host PodAntiAffinity configured
• Container probes, both readiness and liveness checks should be configured, and should not be identical
• Container securityContext, run as high number user/group, do not run as root or with privileged root fs
• Stable APIs, use a stable API if available (supported: Deployments, StatefulSets, DaemonSet)

More: https://github.com/zegl/kube-score/blob/master/README_CHECKS.md

Cluster autoscaling

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-the-key-best-practices-for-running-cluster-autoscaler

Clarify "Liveness probes values aren't the same as the Readiness"

In Liveness probes values aren't the same as the Readiness:

• The title suggests "values"
• The text body speaks about "endpoint":

When Liveness and Readiness probes are pointing to the same endpoint, the effects of the probes are combined.

I think the title is correct, and the task body should be updated to reflect that it's generally fine to have the same endpoint called from liveness and readiness probes, as long as the values are different (and those for liveness are more tolerant than those for readiness).

Referenced article Liveness probes are dangerous:

if you use Liveness Probe, don’t set the same specification for Liveness and Readiness Probe

• you can use a Liveness Probe with the same health check, but a higher failureThreshold (e.g. mark as not-ready after 3 attempts and fail Liveness Probe after 10 attempts)

Cluster must haves

@ipedrazas has just presented at Cloud Native Wales and he has a "must-have" list which looks quite interesting.

We might need to borrow a few items.

Slides

Secrets and configs

“Injected environment variables are always present and may become artifacts in logs for the entire system”
Secret-based environment variables should be sourced from a separate file (volume) in such a way they’re only available to the desired process/container. Not the whole pod.

Source: https://twitter.com/jeyfelbrandauer/status/1194752211366088704