PreStart and PostStop event hooks about kubernetes

[n4j]

@bgrant0607 This issue is open since long is it ok if we close this?

[bgrant0607]

Docker issues requesting postcreate hooks:
moby/moby#3317
moby/moby#252

One of these mentioned that Docker planned to support hooks for every event, but that was quite a while ago, so I'm not sure whether that's still in plan.

[bgrant0607]

Just got support from tianon for runin/exec/enter support: moby/moby#1228
So we can use that to execute hooks inside containers

[bgrant0607]

I investigated Docker's event stream. It provides container id and event type (e.g., start, stop). It doesn't provide further details about the events. Also, it's obviously asynchronous with respect to the events. At least pre-start and post-termination event hook commands would be most useful if executed synchronously, inline with container execution.

Docker restart allows restarting the same command in the same container with the same container id and filesystem, even after the death of the previous process. There's no way to change the command executed AFAICT, however. If the forthcoming "runin"/"exec" allowed execution in dead containers, we could maybe use it for post-termination hooks. Pre-start hook commands look ugly without Docker support. No response yet to my docker-dev question about whether Docker is actually planning to add hook support.

We could override the container entrypoint with the pre-start hook command and then use "runin" to execute the real entrypoint, but the container's status, wait, restart, etc. would be broken. Creating a new container image that included the pre-start command, actual entrypoint, and post-termination hook might work, but we'd need to carefully propagate arguments, signals, exit status, etc.

The main reason for a post-start hook would be consistency, but it also might be convenient for start actions that don't block the start of the container's entrypoint, such as registration in a third-party discovery service, pushing events to pubsub, etc. "runin" should just work, though there may be a race if the application immediately terminated. Probably it would be useful to serialize execution with respect to later hooks on the same entity.

If we wrapped the application we could intercept SIGTERM in order to execute the pre-termination hook, and just pass the signal on to the application if no hook were specified. Again, we probably want to serialize with respect to the post-termination hook. It might not be super-useful to execute the pre-termination hook in the case that it weren't a planned container stop, but I'm not sure whether it's more natural to execute it prior to the post-termination hook regardless or whether it would be annoying for the pre-termination hook to execute when the application was already dead. OTOH, that case would probably need to be handled since the application could die at any time, including concurrently with the start of the pre-termination hook.

Asynchronous webhooks would be comparatively easy to support. The most difficult issue is what to do about auth. If generated by a command in the container, presumably they could authenticate as the container, so it looks appealing to only support command hooks. However, there are situations where we wouldn't have an obvious container to execute commands in, such as for pod lifecycle hooks or, even more obviously, replicationController and service lifecycle event hooks.

[smarterclayton]

We could override the container entrypoint with the pre-start hook command and then use "runin" to execute the real entrypoint, but the container's status, wait, restart, etc. would be broken. Creating a new container image that included the pre-start command, actual entrypoint, and post-termination hook might work, but we'd need to carefully propagate arguments, signals, exit status, etc.

We tried make this work, but it felt wrong on all sorts of levels and ultimately we felt there were two cases we wanted to handle:

• hooks that either need the container context (and as such executing outside the process namespace would be pointless), or if interrupted by container shutdown would not be internally inconsistent. Pre-termination is a good example
• hooks that should be outside of a container, because they need to continue to run even if a container fails. Deploy across multiple containers is a good example, or post-termination.

I don't think you need to subclass the entrypoint in the image - for the former, you could use "docker exec in" (add process to namespace) and for the latter, you can use restart-once containers (outside or inside a pod).

There's some crossover here between "event hooks" and "intent/application hooks" - reacting to changes to the infrastructure, vs reacting to user intent across multiple pods. The "intent hooks" tend to be things that I think of as wanting the auth/orchestration parts of the downward-facing API (I want to wait for this label query to reach X instances, and then update another replication controller to go to Y), as well as being more like regular jobs in their own right.

[thockin]

Is there more to do here, or should we close this and open new issues?

[bgrant0607]

I was waiting for #1445 to go in before closing this one.

[bgrant0607]

Removed from milestone and repurposed for PreStart and PostStop hooks, which I've always thought would be the most useful hooks to provide, for reasons discussed above and in several other issues.

Hooks are being discussed for Docker:
moby/moby#6982

Our users' hooks should execute within the container context. The case for PreStart executing within the container context is to initialize the filesystem, and the case for PostStop is extracting data from the filesystem.

What's attractive about Docker executing the hooks in the host context is that would be provide a hook for the management layer -- Kubelet in our case. The people who advocated executing them in the host context said that nsinit/exec could be then used to enter the container in the hook. That SGTM, since it would give us more control (e.g., allowing us to use Docker container restarts). It would still require the same changes to libcontainer and Docker, however, to decouple container lifetime from process lifetime.

/cc @vishh

[bgrant0607]

Proof-of-concept for prestart here: #3067 (comment)

[yifan-gu]

Trying to implement the PostStart and PreStop hooks we have for rkt. As each app currently is essentially a systemd service within the nspawn container, so I'd like to take advantage of the systemd's service lifecycles. However it only has ExecStartPre, ExecStartPost and ExecStopPost. So I wonder do we really need the hooks to run exactly before the container stops?
@bgrant0607

/cc @dchen1107 @yujuhong @timstclair @vishh @jonboulle

[dalanlan]

Would love to get my hands dirty for PreStart hook, and do you still stick with the idea of #3067(comment)?

/cc @bgrant0607 @vishh @vmarmol

[smarterclayton]

PreStop is a tough one - all the use cases folks have brought up so far
involve "I want to do something to my running process to ensure it
gracefully shuts down".

On Thu, Oct 15, 2015 at 9:34 PM, Emma He [email protected] wrote:

Would love to get my hands dirty for PreStart hook, and do you still
stick with the idea of #3067(comment)
#3067 (comment)
?

/cc @bgrant0607 https://github.com/bgrant0607 @vishh
https://github.com/vishh @vmarmol https://github.com/vmarmol

—
Reply to this email directly or view it on GitHub
#140 (comment)
.

[yujuhong]

PreStop is a tough one - all the use cases folks have brought up so far
involve "I want to do something to my running process to ensure it
gracefully shuts down".

Hasn't that already been covered by the graceful termination you added?

[smarterclayton]

Unfortunately no - there are lots of examples of legacy software that don't
respond to SIGTERM well (Java as an example), so people are writing Prestop
hooks that invoke specific shutdown paths.

On Oct 16, 2015, at 7:55 PM, Yu-Ju Hong [email protected] wrote:

PreStop is a tough one - all the use cases folks have brought up so far
involve "I want to do something to my running process to ensure it
gracefully shuts down".

Hasn't that already been covered by the graceful termination you added?

—
Reply to this email directly or view it on GitHub
#140 (comment)
.

[bgrant0607]

@yifan-gu See also discussion in moby/moby#6982. I'd like the hooks at the container layer to be executed in the host context, so that Kubelet can implement whatever functionality it needs.

@dalanlan Do you have a specific use case for the prestart hook in mind? It's pretty invasive. We may have better alternatives once Docker integrates runc. I think there's more demand for #831.

[dalanlan]

@bgrant0607 Agreed. runc fits much better:)
Our humble use case is, given a container (let's say a tomcat inclusive), we want to inject some upstart param to its java binary to enable our application agents to collect some app-level runtime information. That said, we cannot get there via #831, i'm afraid.

[resouer]

And for now, we have to hack it like this:

1. Inspect Entrypoint + CMD of the original image, let's say: sh -c "tomcat start"
2. In pod.yml, we start the container with: ./configure.sh && tomcat star

[linzichang]

@bgrant0607 @dalanlan I would like to do PreStart hook too. I have implemented PreStart, PostStop and PostRm with an agent, outside kubernetes and docker. PostStop and PostRm are easy to achieve using docker events. Assuming what I want to do in PreStart is running prestart.sh and the real cmd is mycmd. There are two method to implement PreStart.

1. docker run sh -c prestart.sh;mycmd
2. docker run agentexec prestart.sh mycmd
   The agentexec first run the prestart.sh, then use an exec syscall to run mycmd. This make mycmd still be Pid 1.

[bgrant0607]

@linzichang Last I checked, PostStop doesn't actually work as desired because the mount namespace, and hence all volumes, is destroyed.

For PreStop, see: #3067 (comment)

Not all container images contain shells. We could build a static binary and inject it into the container.

I agree that the hook executor needs to exec the user's app so that it runs as pid 1.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[bgrant0607]

/remove-lifecycle rotten
/lifecycle frozen

[timchenxiaoyu]

how to implement prestart hook ?

[adityajoshi12]

There is no updates on the prestart hook, you can use the init container as an alternative to prestart hook.

[sftim]

Are init containers a suitable replacement for preStart?

For Jobs, you can also use a pod failure policy to abort an entire Pod when a specific container fails. Perhaps extending that to other workload controllers, plus the existing init container support, is enough?

[debu99]

i prefer to make the pod zone awareness with environment variable in preStart, but init container can only share volume