The spot-keeper from bsycorp

spot-keeper's People

Contributors

Stargazers

Watchers

spot-keeper's Issues

Nodes are not cordoned during the last block of the day

Cordon is only done in the replace script, not in the maintain even if our of hours.

Notify pods still on dieing spot nodes if they exceed replacement timeout

Could exec and write to process stdout/stderr before spot node death as an FYI to long-running process that it is about to die. Will make it more obvious why a process fails.

via echo hey > /proc/8/fd/1 or similar

kops-controller unable to load instance group object for spotbuilds instances due to missing kops.k8s.io/instancegroup tag

Captured the errors as below in kops-controller logs:

controller.go:218] controller-runtime/controller "msg"="Reconciler error" 
"error"="unable to load instance group object for node ip-10-215-98-144.ap-southeast-2.compute.internal: 
error identifying node \"ip-10-215-98-144.ap-southeast-2.compute.internal\": 
kops.k8s.io/instancegroup tag not set on instance i-0c33a94b5123b8aa4"  
"controller"="node" "request"={"Namespace":"","Name":"ip-10-215-98-144.ap-southeast-2.compute.internal"}

Consequently, spotbuilds nodes are listed without valid label:

% kubectl get nodes | grep none
ip-10-215-98-136.ap-southeast-2.compute.internal   Ready    <none>   118m   v1.17.0
ip-10-215-98-144.ap-southeast-2.compute.internal   Ready    <none>   93m    v1.17.0
ip-10-215-98-147.ap-southeast-2.compute.internal   Ready    <none>   98m    v1.17.0
ip-10-215-98-153.ap-southeast-2.compute.internal   Ready    <none>   115m   v1.17.0
ip-10-215-98-157.ap-southeast-2.compute.internal   Ready    <none>   104m   v1.17.0
ip-10-215-98-158.ap-southeast-2.compute.internal   Ready    <none>   108m   v1.17.0

Recommend Projects