Drain feature about keel

[rusenask]

Hi,

Interesting, have you checked how k8s stops containers during rolling update? I assumed it's graceful shutdown but it probably has a limit of 10 seconds and after that it just kills it.

I was planning to add two internal hooks:

1. pre-update - this hook could probably do what you want. The idea is that it could call out to a remote service and inform it or wait (block) for response.
2. post-update - inform remote service about update.

These hooks could either be inside the codebase (blocking and nonblocking webhooks) or custom code snippets (probably phase 2).

The problem here is that it quickly grows and soon it will require some kind of management interface & UI.

If pre/post update hooks makes sense for you, I will put up a roadmap this evening.

[sabrehagen]

Pre and post would solve this problem and others, so go with them. What would the hook notification mechanism be?

[rusenask]

Not sure yet, initially I thought it would be like a customizable list of endpoints to call for these webhooks and either wait, stop or proceed based on responses.

From your use-case it looks like the endpoint would be a service pointing to a deployment which will be undergoing an update. Are you running one container (replica) per deployment?

How do you think it should be configurable? Maybe each deployment could have label something like:
keel.sh/preupdateHooks=http://whatever.com/update
Problem here is that the value is limited to 63 characters.

Other option is to make these hooks configurable through the API and make a use of k8s label selectors:
selector: environment in (production, qa) endpoints: http://yourServiceName/shouldUpdate

Disadvantages of the option 2 is that Keel would become stateful :)

[sabrehagen]

Definitely want to keep it stateless. HTTP endpoints as you suggested are the way to go. They should just be an HTTP POST with metadata about the operation.

Have to overcome the 63 chars situation though. Maybe use an annotation?

[rusenask]

Excellent, annotations look exactly what we need here!

[sabrehagen]

This can be implemented using the new webhooks feature.

[rusenask]

Hey, webhooks weren't going to work for you as they were intended as "notifications" after the fact and the actual drain feature was planned to be added through plugins. I have created another notification, webhook:

{
	"name": "update deployment",
	"message": "message here",
	"createdAt": "2017-07-11T09:12:56.476274094+01:00",
	"type": "preparing deployment update",
	"level": "LevelDebug"
}

You can block now on that LevelDebug one and ignore others.

Slack :

Hope that helps!

[rusenask]

Closing this due to age, if advanced implementation of this required, please reopen or create a new one.