K3S startup stuck in a deadlock when a KMS provider is configured and the node is rebooted about k3s

[brandond]

The KMS provider itself runs as a pod on the cluster.

I'm not familiar with this deployment pattern for KMS providers - why are you trying to do this? It suffers from the obvious chicken-and-egg problem you're running into here, where the cluster can't start because it needs access to something that won't be available until after it's up.

You're trying to figure out how to lock your keys in the car but still open the door. I don't think there's a good way to make this work.

[jirenugo]

The KMS provider itself runs as a pod on the cluster.

This is not an uncommon pattern for KMS deployment.
Arguably k3s has a circular dependency on kubernetes secrets. It is unfortunate that this is not part of the conformance tests, at least as far as I can tell.

https://github.com/kubernetes-sigs/aws-encryption-provider
https://github.com/Azure/kubernetes-kms?tab=readme-ov-file
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/barbican-kms-plugin/using-barbican-kms-plugin.md
https://github.com/Tencent/tke-kms-plugin/blob/90b71a5c7d78a564567040ebe1ce7135afe99ce5/deployment/tke-kms-plugin.yaml#L4

[brandond]

K3s uses secrets for a couple things internally:

• Node password verification
• Supervisor/apiserver certificate sync

Both of these should soft-fail and retry until secrets can be read. Where exactly does k3s startup stall?

I see that https://github.com/kubernetes-sigs/aws-encryption-provider for example suggests running the KMS as a static pod - are you doing that by placing the pod spec in a file in /var/lib/rancher/k3s/agent/pod-manifests/, or are you trying to deploy it via kubectl apply?

[jirenugo]

suggests running the KMS as a static pod

Yes. Static pods have the same issue.

Both of these should soft-fail and retry until secrets can be read. Where exactly does k3s startup stall?

I don't know. I attached the logs from the systemd service in the issue where it's trying to access /registry/secrets/kube-system/k3s-serving. Does that answer your question? Why does it hard fail on this secret? I can get more logs if you share instructions.