ValidatingAdmissionPolicy: auditAnnotations are included in the audit event always about kubernetes

[hmonsalv]

/sig api-machinery

[seans3]

/triage accepted
/assign @cici37

[cici37]

Hi @hmonsalv Thanks for raising the issue!
VAP is using validation.policy.admission.k8s.io/validation_failure as the AuditAnnotation and it seems working fine for the example you provided.(The VAP Annotation Reference)
From the issue description, "validation.policy.admission.k8s.io/validation_failure" is returned for my-pod-10 and didn't return from my-pod-20 which is expected. prevent-default-namespace/validation_failed is not the annotation used for ValidatingAdmissionPolicy. Thanks

[hmonsalv]

Hi @cici37, thanks for you answer!

I have edited the VAP definition of the example to reproduce the isse, I committed a mistake in the VAP definition that maybe created some confusion.

The default audit annotation validation.policy.admission.k8s.io/validation_failure is working fine as you commented and is only set for the pod violating the VAP my-pod-10 .

But if you take a look 👀 again to the VAP definition prevent-default-namespace defined above, you can see I defined a custom audit annotation prevent-default-namespace/validation_failed with a static value "true":

...
  auditAnnotations:
    - key: "validation_failed"
      valueExpression: "'true'"
...

And this custom annotation I would expect that behaves exactly in the same way as the default one, and is only set if the VAP is violated. But as you can see in these audit events this is not the case, and it's set in the event log corresponding to the creation of both pods (no matter if the VAP was violated or not):

❯ talosctl read /var/log/audit/kube/kube-apiserver.log | grep "my-pod-10" | grep "validation_failed"
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7703d71a-00e8-4b81-9fb2-d08fbd136fc5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods?fieldManager=kubectl-run","verb":"create","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["192.168.65.1"],"userAgent":"kubectl/v1.28.0 (darwin/arm64) kubernetes/855e7c4","objectRef":{"resource":"pods","namespace":"default","name":"my-pod-10","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-15T15:14:40.823489Z","stageTimestamp":"2024-06-15T15:14:40.830217Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}","mutation.webhook.admission.k8s.io/round_0_index_4":"{\"configuration\":\"opentelemetry-operator-mutation\",\"webhook\":\"mpod.kb.io\",\"mutated\":false}","pod-security.kubernetes.io/enforce-policy":"privileged:latest","prevent-default-namespace/validation_failed":"true","validation.policy.admission.k8s.io/validation_failure":"[{\"message\":\"Pods should not be created in the default namespace\",\"policy\":\"prevent-default-namespace\",\"binding\":\"prevent-default-namespace\",\"expressionIndex\":0,\"validationActions\":[\"Warn\",\"Audit\"]}]"}}

❯ talosctl read /var/log/audit/kube/kube-apiserver.log | grep "my-pod-20" | grep "validation_failed"
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"7ba57b1d-6cbd-489d-bef1-709f337b9eda","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/testing/pods?fieldManager=kubectl-run","verb":"create","user":{"username":"admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["192.168.65.1"],"userAgent":"kubectl/v1.28.0 (darwin/arm64) kubernetes/855e7c4","objectRef":{"resource":"pods","namespace":"testing","name":"my-pod-20","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-06-15T15:15:49.607569Z","stageTimestamp":"2024-06-15T15:15:49.614769Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}","mutation.webhook.admission.k8s.io/round_0_index_4":"{\"configuration\":\"opentelemetry-operator-mutation\",\"webhook\":\"mpod.kb.io\",\"mutated\":false}","pod-security.kubernetes.io/enforce-policy":"privileged:latest","prevent-default-namespace/validation_failed":"true"}}

So I really do think there is kind of a bug with custom audit annotations.

Food for another issue / proposal...
I wanted to use this custom annotation in all my VAP, because that way I would get an annotation in the event log for each VAP violated name_of_the_vap/validation_failed, as apparently it's actually the only way to know if the action of an apiserver http request is violating more than one VAP, because currently, as event annotations cannot be overwritten, if more than one VAP are violated, the first one will set the default annotation, and other VAP/s that are violated will fail to overwrite the value of the default annotation, so that you will only be able to see in the event log the default annotation validation.policy.admission.k8s.io/validation_failure with the value (message + VAP + VAP binding) corresponding to the first VAP that was violated. As you can see in the apiserver logs admission annotations are not allowd to be overwritten (typo in allowd btw):

W0619 07:11:45.051058       1 dispatcher.go:320] Failed to set admission audit annotation validation.policy.admission.k8s.io/validation_failure to [REDACTED]  admission annotations are not allowd to be overwritten, key:"validation.policy.admission.k8s.io/validation_failure", old value: [REDACTED]

But I think this is food for a different issue/proposal that I will open soon, don't want to mix topics in this 🧵