`Client.Timeout exceeded` (30s) on validation webhooks when updating Ingress objects about ingress-nginx

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

Is it possible for you to get logs and events from those 20-30 secnds

[longwuyuan]

/remove-kind bug
/triage needs-information

[torvald]

torvald@surdeig ~ $ date; time kubectl patch ing tech-radar-auth --type='json' -p='[{"op": "add", "path": "/metadata/labels/testing", "value": "testing"}]'; date;
Mon Apr 15 09:29:26 AM CEST 2024
ingress.networking.k8s.io/tech-radar-auth patched

real	0m19.327s
user	0m0.350s
sys	0m0.067s
Mon Apr 15 09:29:45 AM CEST 2024

Excluding lines containing:

• "remote_addr" (pattern not to include pure request logs and client IP addresses)
• "Endpoint"
• "body is buffered"
• "Error getting SSL certificate"
• "Unexpected error validating SSL certificate"
• "does not contain a Common Name"

I'm left with these for the time period as mention above + 15 sec (09:29:26 - 09:29:59):

2024-04-15 09:29:45.477	admission.go:149] processed ingress via admission controller {testedIngressLength:269 testedIngressTime:18.855s renderingIngressLength:269 renderingIngressTime:0.002s admissionTime:9.5MBs testedConfigurationSize:18.857}
2024-04-15 09:29:45.478	main.go:107] "successfully validated configuration, accepting" ingress="infra/tech-radar-auth"
2024-04-15 09:29:45.486	event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"infra", Name:"tech-radar-auth", UID:"e635a88c-e72e-401f-8a9a-afa536b06c11", APIVersion:"networking.k8s.io/v1", ResourceVersion:"3624067364", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
2024-04-15 09:29:45.486	event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"infra", Name:"tech-radar-auth", UID:"e635a88c-e72e-401f-8a9a-afa536b06c11", APIVersion:"networking.k8s.io/v1", ResourceVersion:"3624067364", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
2024-04-15 09:29:45.487	event.go:298] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"infra", Name:"tech-radar-auth", UID:"e635a88c-e72e-401f-8a9a-afa536b06c11", APIVersion:"networking.k8s.io/v1", ResourceVersion:"3624067364", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

But for completeness sake, here is the full dump (redacted for business info, and all request logs are excluded): nginx-logs-ing-update-2024-04-15 09 33 02.414.txt

[longwuyuan]

• Does the node on which the controller pods are scheduled see any resource crunch
• Have you configured limits and resources on the pods

There are lots of ssl errors and if you are saying that you submit lots of new ingress json to the api-server and the controller pod needs to reconcile & sync while there is resource crunch, then the stats could be the impact

[torvald]

• Does the node on which the controller pods are scheduled see any resource crunch

No, we have fairly low utilization on all node. These are the three nodes from the same point (marked with the arrows) in time as the previous mentioned test. CPU to the left, memory to the right.

• Have you configured limits and resources on the pods

Yes, full deployment manifest here.

        resources:
          requests:
            cpu: 5000m
            memory: 8000Mi
          limits:
            memory: 8000Mi

There are lots of ssl errors and if you are saying that you submit lots of new ingress json to the api-server and the controller pod needs to reconcile & sync while there is resource crunch, then the stats could be the impact

Events that make nginx sync data can be seen here, my test from earlier is marked, a period of low «sync volume».

Yes, there are a unfortunate amount of SSL errors. Zooming out, these are rather constant and I don't feel like they should affect the validation of my config to the point it take 20s. Following the same argument, there should be periods of low resource crunch where we should see validation duration of only a few seconds, but we don't.

I can try and eliminate the SSL errors before going further to get rid of this suspicion.