xunholy / k8s-gitops

Kubernetes cluster powered by GitOps with FluxCD- Unified source of truth, automated workflows, declarative infrastructure, and cutting-edge DevOps practices.

Raspbernetes

My Personal Kubernetes GitOps Repository

... managed with Flux, Renovate and GitHub Actions

🍼 Overview

This educational project is designed to provide a hands-on learning experience for mastering Kubernetes cluster configurations and best practices. The repository showcases a declarative implementation of a Kubernetes cluster, following GitOps principles that can be utilized with a variety of tools and workflows.

The main goal of this project is to demonstrate best practices for implementing enterprise-grade security, observability, and comprehensive cluster configuration management using GitOps in a Kubernetes environment, while fostering learning and growth in the Kubernetes community.

This repository leverages a range of cutting-edge open-source tools and platforms, forming a comprehensive technology stack that demonstrates the power of the CNCF ecosystem.

📖 Table of contents

• 🍼 Overview
  • 📖 Table of contents
  • 📚 Documentation
  • 📋 Requirements
  • 🚀 Quick Start
  • 🔧 Hardware
  • ☁️ Cloud Services
  • 🖥️ Technology Stack
  • 🤖 Automation
  • 🤝 Acknowledgments
  • 👥 Contributing
    • 🚫 Code of Conduct
    • 💡 Reporting Issues and Requesting Features
  • 📄 License

📚 Documentation

• Raspbernetes Docs
• Kubernetes @Home Docs

📋 Requirements

In order to effectively utilize this repository, it is important to have the following tools set up in your environment.

• Kubernetes cluster
• Flux installed
• Kustomize installed
• Taskfile installed

🚀 Quick Start

1. Set up the necessary environment variables:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>
export GITHUB_REPO=<your-repo>
export CLUSTER=<target-cluster>

2. Verify that your cluster satisfies the prerequisites:

flux check --pre

3. Run the bootstrap command to install Flux and deploy into the cluster:

task cluster CLUSTER_NAME=cluster-1

Note: Many variables in the cluster depend on your specific configuration and should be modified accordingly. Be sure to review and adjust these variables as needed to match your environment and requirements.

🔧 Hardware

Device	Description	Quantity	CPU	RAM	Architecture	Operating System	Notes
Protectli FW6E	Router	1	4 Cores	16GB RAM	AMD64	VyOs
Protectli VP2410	Kubernetes Control Plane	3	4 Cores	8GB RAM	AMD64	Talos Linux
Protectli FW2B	Kubernetes Node(s)	3	2 Cores	8GB RAM	AMD64	Talos Linux
Raspberry Pi 4 Model B	Kubernetes Node(s)	4	4 Cores	8GB RAM	ARM64	Talos Linux
Rock Pi 4 Model C	Kubernetes Node(s)	6	4 Cores	4GB RAM	ARM64	Talos Linux

☁️ Cloud Services

Although I manage most of my infrastructure and workloads on my own, there are specific components of my setup that rely on cloud services.

Service	Description	Cost (AUD)
Cloudflare	I use Cloudflare in my home network for DNS management and to secure my domain with Cloudflare's services.	~$69/yr
GCP	I use Google Cloud Platform (GCP) to manage backups using Google Cloud Storage (GCS) and employ GCP's OAuth for authentication.	~20/mo
GitHub	I use GitHub for code management and version control, enabling seamless collaboration in addition to OAuth for authentication	Free
NextDNS	I use NextDNS for malware protection and ad-blocking for a safer browsing experience.	~$30/yr
UptimeRobot	I use UptimeRobot to monitor my home services for uninterrupted performance.	~$84/yr
Lets Encrypt	I use Let's Encrypt to generate certificates for secure communication within my network.	Free
		Total: ~$35/mo

🖥️ Technology Stack

The below showcases the collection of open-source solutions currently implemented in the cluster. Each of these components has been meticulously documented, and their deployment is managed using FluxCD, which adheres to GitOps principles.

The Cloud Native Computing Foundation (CNCF) has played a crucial role in the development and popularization of many of these tools, driving the adoption of cloud-native technologies and enabling projects like this one to thrive.

	Name	Description
	Kubernetes	An open-source system for automating deployment, scaling, and management of containerized applications
	FluxCD	GitOps tool for deploying applications to Kubernetes
	Talos Linux	Talos Linux is Linux designed for Kubernetes
	Cilium	GitOps tool for deploying applications to Kubernetes
	Istio	Istio extends Kubernetes to establish a programmable, application-aware network using the powerful Envoy service proxy.
	containerd	Container runtime integrated with Talos Linux
	CoreDNS	A DNS server that operates via chained plugins
	MetalLB	Load-balancer implementation for bare metal Kubernetes clusters, using standard routing protocols.
	Prometheus	Monitoring system and time series database
	Jaeger	Open-source, end-to-end distributed tracing for monitoring and troubleshooting transactions in complex distributed systems
	Helm	The Kubernetes package manager
	Falco	Container-native runtime security
	Flagger	Progressive delivery Kubernetes operator (Canary, A/B Testing and Blue/Green deployments)
	Open Policy Agent	An open-source, general-purpose policy engine
	Kyverno	Kubernetes Native Policy Management
	Dex	An identity service that uses OpenID Connect to drive authentication for other apps
	Crossplane	Manage any infrastructure your application needs directly from Kubernetes
	Litmus	Chaos engineering for your Kubernetes
	OpenEBS	Container-attached storage
	OpenTelemetry	Making robust, portable telemetry a built in feature of cloud-native software.
	Thanos	Highly available Prometheus setup with long-term storage capabilities
	Cert Manager	X.509 certificate management for Kubernetes
	Grafana	Analytics & monitoring solution for every database.
	Loki	Horizontally-scalable, highly-available, multi-tenant log aggregation system
	Velero	Backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes.

🤖 Automation

This repository is automatically managed by Renovate. Renovate will keep all of the container images within this repository up to date automatically. It can also be configured to keep Helm chart dependencies up to date as well.

🤝 Acknowledgments

A special thank you to everyone in the Kubernetes @Home Discord community for their valuable contributions and time. Much of the inspiration for my cluster comes from fellow enthusiasts who have shared their own clusters under the k8s-at-home GitHub topic.

Also I extend heartfelt thanks to all CNCF contributors for their dedication and expertise, as their collective efforts have been vital in driving innovation and success within the cloud-native ecosystem.

For more ideas on deploying applications or discovering new possibilities, be sure to explore the Kubernetes @Home search search.

👥 Contributing

Our project welcomes contributions from any member of our community. To get started contributing, please see our Contributor Guide.

🚫 Code of Conduct

By participating in this project, you are expected to uphold the project's Code of Conduct. Please report any unacceptable behavior to the repository maintainer.

💡 Reporting Issues and Requesting Features

If you encounter any issues or would like to request new features, please create an issue on the repository's issue tracker. When reporting issues, include as much information as possible, such as error messages, logs, and steps to reproduce the issue.

Thank you for your interest in contributing to this project! Your contributions help make it better for everyone.

📄 License

This repository is Apache 2.0 licensed

k8s-gitops's Issues

CIS Kubernetes Benchmark v1.5.1 #1.2.2

Details

Describe the solution you'd like:

Ensure that the --basic-auth-file argument is not set

Anything else you would like to add:

Audit:
Run the following command on the master node:

ps -ef | grep kube-apiserver

Verify that the --basic-auth-file argument does not exist.

Remediation:

Follow the documentation and configure alternate mechanisms for authentication. Then, edit the API server pod specification file /etc/kubernetes/manifests/kube- apiserver.yaml on the master node and remove the --basic-auth-file= parameter.

Enable FluxCD to integrate with git-crypt to apply existing sealed secret private key

Details

The sealed secret operator pod starts will generate a new private key if one does not exist within the cluster. Currently, the private key that is used to decrypt all existing secrets is stored in git-crypt and manually applied post-setup of Flux in the cluster using the install/sealed-secret.sh script.

Ideally, Flux being integrated with git-crypt means Flux will have the capability to check out the private key within the cluster and apply it without requiring manual intervention after the operator has already initialized it.

https://github.com/bitnami-labs/sealed-secrets#managing-existing-secrets

Grafana admin permissions to users logged in via oauth2

Details

No user currently has admin permissions to Grafana. Users logged in via oauth2 should be made admins.

Authorization policies for L7 mesh connectivity

Details

We should be able to run authorization filter for L7 (Envoy) proxies within the mesh to check if the incoming request is authorized or not.

More info for decision making https://blog.openpolicyagent.org/envoy-external-authorization-with-opa-578213ed567c

CIS Kubernetes Benchmark 1.5.1 # 5.4

5.4 Secrets Management

• 5.4.1 Prefer using secrets as files over secrets as environment variables
• 5.4.2 Consider external secret storage

CIS Kubernetes Benchmark v1.5.1 # 2

Details

2 etcd

• 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate
• 2.2 Ensure that the --client-cert-auth argument is set to true
• 2.3 Ensure that the --auto-tls argument is not set to true
• 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate
• 2.5 Ensure that the --peer-client-cert-auth argument is set to true
• 2.6 Ensure that the --peer-auto-tls argument is not set to true
• 2.7 Ensure that a unique Certificate Authority is used for etcd

Investigate using Fabrikate to hydrate helm manifests

Details

I would like to have my fully hydrated k8s resource manifests checked directly into source code rather than having helm-operator to the template and apply the manifests in-cluster. It also would make version controlling potential breaking changes and making resource changes more clear to observe during the PR process rather than introducing breaking changes directly in-cluster.

https://github.com/microsoft/fabrikate

Setup AuthN for services with no native login

Details

Services such as Prometheus don't have native login oauth2 integration, this can be provided using envoy filters similar to the kiali envoy filter to enforce traffic is authenticated prior to hitting the downstream service via the virtual service resource.

Ability to create cluster using vagrant

Details

Using vagrant and VirtualBox so that anyone should be able to spin up a cluster

Setup Keycloak + oauth2 for nginx ingress

Details

oauth2 proxy is currently setup with github and would be good to also setup with keycloak internal to the cluster.

https://github.com/keycloak/keycloak

https://github.com/codecentric/helm-charts

The image is currently not supported for arm64

Implement SOPS to manage securely storing secrets in source control

Details

Currently, git-crypt is being used within the repository and works nicely, however, there is no visibility into the secret structure of the resource so users who fork the repo don't know what key pair values are required in these secret objects without having to inspect each sealed secret respectively.

SOPs provides encryption similar to git-crypt whilst allowing the structure to be viewable and providing a DIFF in the pull request process.

FluxCD to use either existing ssh keys or Git over HTTPS

Details

Either provide an SSH key which is encrypted in the repo and applied via Ansible or similar approach with Git over HTTPS

https://docs.fluxcd.io/en/1.18.0/guides/provide-own-ssh-key.html
https://docs.fluxcd.io/en/1.18.0/guides/use-git-https.html

Automated security-policies updates

Details

A workflow to update the git sub-module should be setup to automatically promote new policy releases into the repository. Currently, this is a manual process to keep it in parity with the upstream repository.

CIS Kubernetes Benchmark v1.5.1 #1.2.1

Details

Ensure that the --anonymous-auth argument is set to false

Describe the solution you'd like:
Audit:

ps -ef | grep kube-apiserver

Verify that the --anonymous-auth argument is set to false.

Anything else you would like to add:

Remediation:
Edit the API server pod specification file /etc/kubernetes/manifests/kube- apiserver.yaml on the master node and set the below parameter.

--anonymous-auth=false

Integrate cloudflare exporter into prometheus

Details

https://github.com/wehkamp/docker-prometheus-cloudflare-exporter

Run Calico with an eBPF dataplane

Details

v3.16.x Calico eBPF support went GA.

Might need to investigate BGP routing in place of using metallb #75

https://www.projectcalico.org/whats-new-in-calico-3-16/

Remove old dashboards

Details

Revisit the list of currently installed dashboard and remove the ones that are outdated or not in use e.g. Weave Flux

Integrate cluster services into the mesh

Details

Currently, istio has no active workloads deployed into the mesh, this issue would require that most services be integrated and tested whilst running in the mesh using mTLS and no direct external internet facing traffic other than via the egress gateway.

Namespaces to add to the mesh:

• actions-runner-system
• backups
• flux-system
• gatekeeper-system
• home-system
• istio-operator - Should not be added to the mesh
• istio-system - Should not be added to the mesh
• kube-system - Should not be added to the mesh
• litmus
• network
• observability
• openebs
• security

Backup Loki logs to GCS

Details

Store loki logs remotely in GCP

https://github.com/grafana/loki/blob/master/docs/sources/configuration/examples.md#google-cloud-storage

Sealed secrets validation

Details

Describe the solution you'd like:

Sealed secrets must be encrypted with the appropriate public key and information to be decrypted in the cluster. These can cause issues when not valid or have been misconfigured, testing should be done to ensure these are not invalidated due to the before mentioned reasons.

To accomplish this there is a kubeseal --validate command that will --dry-run a decryption against the running sealed secret controller to ensure it is able to decrypt appropriately, if the resources is not valid it will prompt the following error output error: unable to decrypt sealed secret

Migrate to fluxv2

Details

Moving to Fluxv2 will require a massive uplift.

Currently, there is still a dependency on upstream providing arm64 images, and a means to deploy using those images as GHCR doesn't support multi-arch images and therefore images are appended with -arch EG: -arm64 also the default installation using gotk sets amd64 node affinity.

fluxcd/flux2#194

Update MetalLB chart & remove hardcode references to tags

New Chart location: https://github.com/bitnami/charts/tree/master/bitnami/metallb

https://github.com/raspbernetes/k8s-gitops/blob/ef405eb40bab75e3d826c6f2a8a69de3de4fb7a2/namespaces/network/metallb/metallb.yaml#L20

Deploy helm-operator via fluxcd

Details

What steps did you take and what happened:

Attempted to deploy helm-operator and its required CRD's however Fluxcd did not apply the resources?

What did you expect to happen:

Fluxcd should be able to deploy and manage the helm-operator within the cluster.

Anything else you would like to add:

Note: Miscellaneous information that will assist in solving the issue.

Additional Information:

Note: Anything to give further context to the bug report.

Trial Using Thanos in prom-operator

Details

Configure prometheus operator to run using Thanos

https://github.com/helm/charts/tree/master/stable/prometheus-operator

Use terraform to spin up a cluster

Details

Create a terraform module so that users will be able to spin up clusters and get involved in the project.

Troubleshoot/test Velero recovery scenario

Details

Velero operator has been configured in the cluster, backing up resources & snapshots. Create playbook to that tests the recovery scenario.

Deploy Gatekeeper with REGO policies

Details

Deploying Gatekeeper into the cluster with the CIS benchmark policies written in the raspbernetes/k8s-security-policies repository.

Automate CNAME records in cloudflare for new Ingress resources

Details

Due to not having a wildcard CNAME currently, all CNAMEs are manually created in the cloudflare console, ideally though whenever either an ingress or equivalent object is created the CNAMEs should be updated to include the new object automatically and likewise once an object is removed it should no longer persist as a CNAME in cloudflare.

EDIT** Wildcard CNAME records were available previously within cloudflares initial offering, however, it has since become an enterprise function which cost is unjustifiable for a homelab.

Istio third party JWT

Details

https://istio.io/latest/docs/ops/best-practices/security/#configure-third-party-service-account-tokens

2020-07-23T03:55:07.799195Z	info	Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.

Adopt chaos engineering principles by leveraging litmus.

Details

Adopt chaos engineering principles by leveraging litmus. This would be a fantastic way to test and validate the clusters' resilience and recovery with configurable levels of disruption.

Currently pending arm64 support: litmuschaos/litmus#1550

Migrate to new prometheus operator helm charts

Details

The existing helm chart is being deprecated and has moved to the new chart location.

Old: https://github.com/helm/charts/tree/master/stable/prometheus-operator
New: https://github.com/prometheus-community/helm-charts

Configure istio ingress to replace nginx ingress

Details

Implement istio to allow ingress to the cluster and eventually phase out using nginx ingress. Some caveats may be oauth2 and keycloak integration not readily available in istio natively and may require further investigation - might run istio on a subdomain as a proof of concept before migrating completely across.

Setup alert manager against rule sets

Details

Configure alert manager with some sample/basic rule sets.

Enable pull thru cache using containerd

Details

TBC

Monitoring for Fluxv2 with prometheus & grafana

Details

Setup Dashboard for Fluxv2 (Weave Flux is not in use/outdated).

Implement cert-manager monitoring

Details

Implement a service monitor for cert-manager and configure the Grafana dashboard. This may be possible within the chart itself but needs further investigation

Dashboard examples:

• https://grafana.com/grafana/dashboards/11001
• https://grafana.com/grafana/dashboards/12170

Update network policies and global network policies

Details

Network policy currently allows all traffic, these need to be updated to allow isolated network traffic between namespaces and workloads respectively within the internal cluster network traffic.

Where feasible and not overlapping with istio authorization policies, global network policies provided via either Calico or Cilium should be enforced to provide L7 network controls.

CIS Kubernetes Benchmark 1.5.1 # 5.5

5.5 Extensible Admission Control

• 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller

Deploy istio add-ons outside of istio installation

Details

In istio 1.8.x add-on installation via istioctl will no longer be embedded and available as they're deprecated in 1.7.x and removed in 1.8.x

These services will need to be deployed via their respective deployment methods as standalone workloads

https://istio.io/latest/blog/2020/addon-rework/#timeline

CIS Kubernetes Benchmark v1.5.1 #1.1.1

Details

Describe the solution you'd like:

Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Scored)

Anything else you would like to add:

Audit:
Run the below command (based on the file location on your system) on the master node. For example:

stat -c %a /etc/kubernetes/manifests/kube-apiserver.yaml

Verify that the permissions are 644 or more restrictive.

Remediation:
Run the below command (based on the file location on your system) on the master node. For example:

chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml

Use TF to create required resources in GCP

Details

The following are resources within GCP that are required dependencies within this project:

• Velero:
  • GCS
  • GSA
  • Custom Role
• Thanos:
  • GCS
  • GSA
  • Custom Role
• SOPS
  • KMS
  • GSA
  • Custom Role

This would also include things such as enabling required APIs and can further expand into other cloud equivalent services.

helm-operator will be templated via CI and commit to PR on upstream release

Details

Describe the solution you'd like:

When the upstream helm charts or maintainers provide a new official release the helm-operator should be able to be templated via CI and create a PR with the proposed upgrade within the repository.

Anything else you would like to add:

It could even be possible that the particular PR is using Flagger to test the upgrade before complete rollout

Additional Information:
N/A

Migrate branch master to main

Details

Migrate master branch to the main branch; GHA action workflows and repo.yaml will need references updated.

Implement Cilium with eBPF

Details

https://github.com/cilium/cilium

Enable canary helm releases with Flagger in GitOps

Details

Describe the solution you'd like:

Enable the ability to canary out certain functionality and changes using a tool such as Flagger

https://github.com/weaveworks/flagger

Anything else you would like to add:

N/A

Additional Information:

N/A

Automate releases based on scheduled upgrades and/or tags

Details

Implement an auto-release mechanism that can publish releases with changes that have been deployed into the cluster, this would also be handy to version pin the cluster to a given SHA and allowing newly created functionality continue to be merged into master/main branch.

One such tool that could help facilitate this functionality could be https://github.com/marketplace/actions/release-drafter

Update submodule repo name

Details

Update the submodule with the correct repository link, the repository was renamed and Github will continue to redirect the link until a new repository with the same name collides with it in which it will no longer work.

This should be updated so there is a consistent behavior.

Investigate gVisor

Details

https://github.com/google/gvisor

Note: Currently supports ARM64

Integrate Grafana auth with Github login

Details

Rather than users required to use basic auth to login to the grafana instance use github auth to allow authenticated logins.

helm/charts#22473

Currently pending the above issue being implemented otherwise may need to consider another approach.

https://github.com/raspbernetes/k8s-gitops/blob/ef405eb40bab75e3d826c6f2a8a69de3de4fb7a2/namespaces/observability/prometheus-operator/prometheus-operator.yaml#L115

CIS Kubernetes Benchmark v1.5.1 #1.2.1

Details

Ensure that the --anonymous-auth argument is set to false

Describe the solution you'd like:
Audit:

ps -ef | grep kube-apiserver

Verify that the --anonymous-auth argument is set to false.

Anything else you would like to add:

Remediation:
Edit the API server pod specification file /etc/kubernetes/manifests/kube- apiserver.yaml on the master node and set the below parameter.

--anonymous-auth=false