Pod stuck when pod start error about training-operator

[Syulin7]

IIRC, there is no way to set job status to failed. Perhaps we could add StartingDeadlineSeconds(similar to ActiveDeadlineSeconds.) in runPolic, which specifies the time a job can remain Created. If the job is not running by the end of this period, it would be set to failed.

[tenzen-y]

@hywell-h Did you try using backoffLimit?

[hywell-h]

@hywell-h Did you try using backoffLimit?

when pod gose into ImagePullBackOff , the status.phase of pod is Pending， and restart count of pod is zero;So backoffLimit will not work, because pod has not reached the Completed or Failed state yet.

[tenzen-y]

@hywell-h Did you try using backoffLimit?

when pod gose into ImagePullBackOff , the status.phase of pod is Pending， and restart count of pod is zero;So backoffLimit will not work, because pod has not reached the Completed or Failed state yet.

Oh, that makes sense.
Did you try using ActiveDeadlineSeconds? But I understand the ActiveDeadlineSeconds wouldn't be resolved the root cause.

To give failure handling more flexibility, I guess that the training-operator needs to migrate to the batch/v1 Job from plain pods management, and then support JobPodFailurePolicy.

[kannon92]

This sounds like the motivation behind kubernetes/kubernetes#122300

[tenzen-y]

This sounds like the motivation behind kubernetes/kubernetes#122300

@kannon92 Yeah, that's right, but as I remember correctly, configurable pod-stuck errors was rejected by SIG-Node, right?

[Syulin7]

Did you try using ActiveDeadlineSeconds? But I understand the ActiveDeadlineSeconds wouldn't be resolved the root cause.

Yes, the pod status is always pending and it does not trigger the ActiveDeadlineSeconds.

[kannon92]

@tenzen-y there was no rejection.

[tenzen-y]

@tenzen-y there was no rejection.

Ah, I see. As I recheck the issue, the next action seems to raise this topic in the SIG Node meeting.
We probably can move this forward in the next cycle since the current is under the enhancement freeze for the 1.30.

[kannon92]

Yea its mostly that I moved to Red Hat and I'm not as focused on batch related items so I don't really have the bandwidth to lead that one.

[tenzen-y]

Yea its mostly that I moved to Red Hat and I'm not as focused on batch related items so I don't really have the bandwidth to lead that one.

No worries. As I'm still there (WG Batch), let me check the enhancement in the next cycle if I can find enough time.