Failure of Kubernetes Cluster Startup Due to `FailurePolicy=Fail` Parameter in Webhook about gatekeeper

[ritazh]

Thanks for reporting this @jthin. Please take a look at https://open-policy-agent.github.io/gatekeeper/website/docs/failing-closed if you plan to configure the FailurePolicy as Fail. There are few considerations, especially https://open-policy-agent.github.io/gatekeeper/website/docs/failing-closed#admission-deadlock which sounds like your scenario. This can normally be mitigated by deleting the ValidatingWebhookConfiguration, per the emergency procedure.

[jthin]

Hello @ritazh,

Thank you for your response.

Based on my understanding, this is an expected behavior.
However, for many reasons, we need to stop several AKS clusters, particularly in dev/test environment. Is deleting the ValidatingWebhookConfiguration the sole solution, considering that we must reapply it afterward?

[ritazh]

@jthin deleting the ValidatingWebhookConfiguration is for emergency to get the cluster back to a healthy state. Disabling the webhook is to ensure cluster operations are not blocked by the webhook when the webhook is down.

To ensure the webhook does not affect your cluster long term when FailurePolicy=Fail, from api server logs, can you tell which resources have been blocked by the webhook when the cluster is restarted?

Few things you may need to update in your ValidatingWebhookConfiguration to reduce the number of resources that must go thru this webhook:

1. Under rules, is the scope set to * or Namespaced?
2. If it's namespaced, are system namesapaces excluded via matchExpressions? e.g. kube-system, istio-system
3. Under resources, are cluster-scoped resources excluded, like nodes? What about ConfigMap as it is used as the locking resource for some Kubernetes leader elections.

There are probably more resources that need to be excluded in ValidatingWebhookConfiguration to ensure cluster operations are not blocked by a failed webhook.

[maxsmythe]

in newer clusters, MatchConditions may also help with scoping:

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#matching-requests-matchconditions

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.