When Pod Disruption Budget blocks Node draining, no restart is possible about kured

[awh]

Kured should ignore this error and reboot the Node anyway

I'm not sure this is the correct default - presumably one has configured a disruption budget for a reason 😄 But certainly we can have it as an option.

I'd rather avoid parsing the output of kubectl drain, so this will have to wait until after we migrate away from embedding kubectl to using the library version of drain.

[antoineozenne]

Same problem for a standard cluster without any service mesh. There are too cases : the pod ("managed" by the PDB) can't be scheluded on another node, so the situation is blocked in this state, or the pod is rescheduled by Kubernetes on another node, but it takes time so Kured timeouts.

[s-spindler]

Faced a similar issue but for me the pod could have easily been rescheduled by k8s on another node. The PDB is quite simple and only states that min 1 pod is running.
@awh I think that it is the correct default. AFAIU when deleting a pod that has a PDB such as mine, k8s will simply make sure that a second pod is running before killing the old one. Maybe I'm not getting PDBs here but my understanding was, that k8s would take care of fulfilling the PDB.
In the end I simply killed the pod by hand and sure enough a second one was spun up before the first one shut down.

[s-spindler]

One more thing: if this isn't changed I would expect kured to at least not set the node to SchedulingDisabled. ATM the node is basically drained of pods except for the ones with PDBs.

[evrardjp]

One more thing: if this isn't changed I would expect kured to at least not set the node to SchedulingDisabled. ATM the node is basically drained of pods except for the ones with PDBs.

Shouldn't drain first step be "cordon"ing the node (which effectively makes it 'SchedulingDisabled')? I thought that was its internal behaviour.

[uncycler]

An other issue related to PDB, when this happended, CPU usage is up to 100% and it fills logs with hundred of error per seconds.

[github-actions]

This issue was automatically considered stale due to lack of activity. Please update it and/or join our slack channels to promote it, before it automatically closes (in 7 days).

[uipo78]

Should this issue be closed?

[kinihun]

Yes, please do. Thanks. From: Branden Carrier <[email protected]> Sent: 22 December 2020 14:46 To: weaveworks/kured <[email protected]> Cc: Olumide <[email protected]>; Author <[email protected]> Subject: Re: [weaveworks/kured] When Pod Disruption Budget blocks Node draining, no restart is possible (#82) Should this issue be closed? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#82 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AELD2R267S6T35VPAC5KCXTSWCWJTANCNFSM4IGRLYOQ> . <https://github.com/notifications/beacon/AELD2RYB7MB6SS2PDV4M2ZLSWCWJTA5CNFSM4IGRLYO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFSWZYVY.gif>