Controller failed to renew lease (leader election) due to timeout until pod restart about ingress-nginx

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[longwuyuan]

/remove-kind bug

• Post output of kubectl get events -A
• Post logs of all the controller pods

Any idea why a re-election was triggered
Any other info at cluster level or on the nodes

/triage needs-information

[satdeveloping]

Post output of kubectl get events -A

All there is now is:

LAST SEEN   TYPE     REASON                OBJECT                                      MESSAGE
4m25s       Normal   UpdatedLoadBalancer   service/ingress-nginx-controller-internal   Updated load balancer with new hosts
4m24s       Normal   UpdatedLoadBalancer   service/ingress-nginx-controller            Updated load balancer with new hosts

Post logs of all the controller pods

There was only one controller pod at the time of the incident.

I tried looking at host logs for where the pod was scheduled. Between 2024-04-18T12:56:14 and 13:07:42 there was only one message:

{
    "host": "ip-172-32-107-61",
    "ident": "rsyslogd",
    "message": "imjournal: 182979 messages lost due to rate-limiting",
    "az": "eu-west-1a",
    "ec2_instance_id": "i-057706eb5156f775d"
}

There were some logs from the kubelet.service that look like it was restarting. It was printing out the configured flags and saying that "such-and-such Service is Starting".

Perhaps everything just got in to a twist?

The controller pod was Running long before and after the time of the incident

[strongjz]

It looks like the controller couldn't reach the API server with all the dial tcp 10.100.0.1:443: i/o timeout

The restart fixed the issue; I'm going to close it for now; if it does happen again, feel free to reopen it.

[strongjz]

/close

[k8s-ci-robot]

@strongjz: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.