Node reboot leaving existing pod using resources stuck with error UnexpectedAdmissionError about kubernetes

[liuyuan10]

While this issue can be easily reproduced with node reboot, I believe it can also happen if device plugin suddenly becomes unhealthy after pod is scheduled and before kubelet allocates the resources.

The expected behavior is that the failure should be retried just like CNI failures.

[tamilselvan1102]

/sig node

[SamDecrock]

I can confirm this issue. Luckily for us the pods are managed by Deployments and they are restarted. But we're still stuck with 'ghost' pods in an UnexpectedAdmissionError state.

[ffromani]

What happened?

When a node is rebooted, pod using resources allocated by device plugin will encounter UnexpectedAdmissionError error as below:

  Warning  UnexpectedAdmissionError  84s                kubelet            Allocate failed due to no healthy devices present; cannot allocate unhealthy devices xxx, which is unexpected

Hi! This behavior wants to make evident and recoverable a previously hidden breakage when pod actually started, but noone allocated the device they requested: #109595

I don't know the mechanics of that specific GPU device plugin, but it seems likely to me that this problem can totally happen also for gpu devices. IOW, from what I gathered it was not device-specific but rather a flaw in how the devices are handled in kubelet, hence the flag.

What makes it really bad is if it's a raw pod, it stucks in such state and never recover.

This behavior can be surprising indeed, but when pod reach terminal state the system doesn't try to recover them, this is why the platform introduce and recommends higher level controllers like deployment

What did you expect to happen?

The pod should be retried until device plugin is ready

Yes, this is something we've being discussed since a while but there's unfortunately nothing concrete yet :\

[ffromani]

/triage accepted

kubelet retries are being discussed since a while and there's general agreement it's a desirable behavior