GithubHelp home page GithubHelp logo

justinas / nixos-ha-kubernetes Goto Github PK

View Code? Open in Web Editor NEW
226.0 226.0 6.0 66 KB

Toy highly-available Kubernetes cluster on NixOS

License: MIT License

Nix 84.77% Shell 2.92% HCL 12.31%
colmena kubernetes kubernetes-cluster nix nixops nixos terraform

nixos-ha-kubernetes's People

Contributors

justinas avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

algobraio starcraft66 gracig urizafrir spiage meta-automata-nix

nixos-ha-kubernetes's Issues

Cert issue with kubernetes-dashboard

And I also tried to add dashboard with this commands:

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/ --kubeconfig ./certs/generated/kubernetes/admin.kubeconfig

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard --kubeconfig ./certs/generated/kubernetes/admin.kubeconfig

Release "kubernetes-dashboard" does not exist. Installing it now.
NAME: kubernetes-dashboard
LAST DEPLOYED: Thu Jun  6 05:47:44 2024
NAMESPACE: kubernetes-dashboard
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*************************************************************************************************
*** PLEASE BE PATIENT: Kubernetes Dashboard may need a few minutes to get up and become ready ***
*************************************************************************************************

Congratulations! You have just installed Kubernetes Dashboard in your cluster.

To access Dashboard run:
  kubectl -n kubernetes-dashboard port-forward svc/kubernetes-dashboard-kong-proxy 8443:443

NOTE: In case port-forward command does not work, make sure that kong service name is correct.
      Check the services in Kubernetes Dashboard namespace using:
        kubectl -n kubernetes-dashboard get svc

Dashboard will be available at:
  https://localhost:8443

And I get this issue:

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ k get pods -o wide -n kubernetes-dashboard
NAME                                                    READY   STATUS             RESTARTS         AGE   IP            NODE      NOMINATED NODE   READINESS GATES
kubernetes-dashboard-api-868878b978-z4l5w               0/1     CrashLoopBackOff   11 (3m11s ago)   34m   10.200.38.3   worker2   <none>           <none>
kubernetes-dashboard-auth-66564b9c9c-xpwxr              1/1     Running            0                34m   10.200.38.5   worker2   <none>           <none>
kubernetes-dashboard-kong-76dff7b666-g64jb              1/1     Running            0                34m   10.200.38.4   worker2   <none>           <none>
kubernetes-dashboard-metrics-scraper-555758b9bf-f5xpf   1/1     Running            0                34m   10.200.5.3    worker1   <none>           <none>
kubernetes-dashboard-web-846f5f49b-2gtgt                1/1     Running            0                34m   10.200.5.4    worker1   <none>           <none>

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ k logs kubernetes-dashboard-api-868878b978-z4l5w -n kubernetes-dashboard
I0606 03:19:28.381701       1 main.go:40] "Starting Kubernetes Dashboard API" version="1.7.0"
I0606 03:19:28.381772       1 init.go:47] Using in-cluster config
E0606 03:19:28.381817       1 config.go:529] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: error creating pool from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: data does not contain any valid RSA or ECDSA certificates
F0606 03:19:28.384162       1 main.go:159] Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get "https://10.32.0.1:443/version": tls: failed to verify certificate: x509: certificate signed by unknown authority
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

Can you please help me?

Best wishes,
Sergei

ter apply fails with can't find storage pool 'default'

following the guide the setup always fails for me with ter apply

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

libvirt_volume.nixos_boot: Creating...
β•·
β”‚ Error: can't find storage pool 'default'
β”‚
β”‚   with libvirt_volume.nixos_boot,
β”‚   on main.tf line 50, in resource "libvirt_volume" "nixos_boot":
β”‚   50: resource "libvirt_volume" "nixos_boot" {
β”‚
β•΅

I made sure that I am in the correct group:

$ groups
users wheel networkmanager libvirtd docker

and that libvirtd is running:

$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/etc/systemd/system/libvirtd.service; enabled; preset: enabled)
    Drop-In: /nix/store/y07n6kr0sk83gqji3lqzykzsrk94f319-system-units/libvirtd.service.d
             └─overrides.conf
     Active: active (running) since Thu 2023-03-23 15:12:51 CET; 14s ago
TriggeredBy: ● libvirtd-ro.socket
             ● libvirtd.socket
             ● libvirtd-admin.socket
       Docs: man:libvirtd(8)
             https://libvirt.org
   Main PID: 5239 (.libvirtd-wrapp)
         IP: 0B in, 0B out
         IO: 0B read, 0B written
      Tasks: 19 (limit: 32768)
     Memory: 5.2M
        CPU: 150ms
     CGroup: /system.slice/libvirtd.service
             └─5239 /nix/store/ziiig351fxfw1rizgnfpbkllxx2xlzvk-libvirt-8.10.0/sbin/libvirtd --config /nix/store/rd9dai9irpxsz9bw98fz47m1wb9zl31g-libvirtd.conf --timeout 120

MΓ€r 23 15:12:51 gestalt systemd[1]: Starting Virtualization daemon...
MΓ€r 23 15:12:51 gestalt systemd[1]: Started Virtualization daemon.

Set up CI (e.g. GitHub Actions)

The CI could at least run ci-lint script inside the nix-shell. There seems to be no way to run virtual machines on GH Actions, so no end-to-end testing can be done.

Bump Flannel to 0.15.1

Flannel 0.15.1 contains a bug fix for the MAC address assignment bug.

nixos-ha-kubernetes/modules/worker/flannel.nix

Lines 44 to 56 in 6442aaa

# https://github.com/flannel-io/flannel/issues/1155
# https://github.com/onixie/nikops/commit/24f66eaa1c6fb40eca4772ee9b933333fe06a85d
#
# Seems to have been fixed in Flannel v0.15.1:
# https://github.com/flannel-io/flannel/releases/tag/v0.15.1
# TODO: try to upgrade flannel package and remove this workaround
systemd.network = {
enable = true;
links."10-flannel" = {
matchConfig.OriginalName = "flannel*";
linkConfig.MACAddressPolicy = "none";
};
};

However, Nixpkgs is still on 0.13.0.

feature request: option for macvlans

It would be amazing if it was possible to use this with macvlans, so that I could treat this cluster as a basically real one within my network.

k top does not work

Hello!

Thank you for this great job!

I'm trying to create environment to learn k8s with your project.
After all steps ./chech.sh give this:

❯ ./check.sh 
https://10.240.0.201:2379, 46916c050e984dc0, 3.5.9, 364 kB, false, false, 2, 370, 370, 
https://10.240.0.216:2379, 96221fa7145e7558, 3.5.9, 360 kB, true, false, 2, 370, 370, 
https://10.240.0.120:2379, db650873350377d7, 3.5.9, 360 kB, false, false, 2, 370, 370, 
Kubernetes control plane is running at https://10.240.0.10

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
uid=0(root) gid=0(root) groups=0(root),10(wheel)
Server:         10.240.0.157
Address:        10.240.0.157:53

** server can't find kubernetes.cluster.local: NXDOMAIN

Name:   kubernetes.default.svc.cluster.local
Address: 10.32.0.1

** server can't find kubernetes.svc.cluster.local: NXDOMAIN

** server can't find kubernetes.cluster.local: NXDOMAIN


** server can't find kubernetes.svc.cluster.local: NXDOMAIN

pod default/busybox terminated (Error)
Success.

So the next steps I tryed was:

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ k top pod
error: Metrics API not available

nixos-ha-kubernetes on  master [!?] via πŸ’  default via οŒ“ impure (nix-shell) 
❯ k top node
error: Metrics API not available

What do I need to do to get the right result?

Best wishes,
Sergei

DNS is not highly-available

Currently, we run one CoreDNS service per worker node (good), but the containers' resolv.conf points only to the host node itself as the name server (bad):

nixos-ha-kubernetes/modules/worker/coredns.nix

Line 39 in f9d01ef

services.kubernetes.kubelet.clusterDns = nodeIP self;

If Kubelet is alive, but CoreDNS is dead on the node, containers running on that node will fail to resolve names.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.