Allow graceful restarts of Verk applications about verk

[edgurgel]

Ok so Verk 1.1.0 has support to pausing/resuming queues! 🎉

Now we just need to work on how to pause queues before shutting down (which should be the easiest part :D)

[edgurgel]

Update:

My plan is:

• Release Verk 1.0.0 with the current master
• Release 1.1.0 with pause and resume #127 + Verk Web release with the interface to pause/resume queues
• Work on a simple way to pause queues when SIGTERM/SIGINT is issued (Erlang 19.3+)

[keyan]

Thanks for getting back. Yeah, I'm happy to poke around some more and see if I can think of something that might work.

Also, seeing as the answer to my question was "no", I updated the issue title to reflect the new goal.

[edgurgel]

I suppose we just need to implement terminate on our managers (workers, scheduler, queue)? I will play with erlang 19.3 later this week. Thanks for the new information!

[seivan]

Yeah, I think that would be a nice design.

Each worker can conform to two protocols (or we call them Behaviour for elixir I guess)
One for doing actual work and one for doing "cancel"

That way each worker gets granular control on how to handle their cancel. Some might just ignore it, others will handle cancel based on what state they're in.

[edgurgel]

I would love to have this feature.

Yeah it would be nice if we could stop the "managers" queue/schedule etc and avoid any changes on the queues.

Do you want to give it a try?

[seivan]

Any progress on this so far?

[seivan]

@edgurgel Just quickly glancing this morning, I'd guess https://github.com/edgurgel/verk/blob/master/lib/verk/queue_supervisor.ex would be a place to trap Unix signals and gracefully manage workers.
My initial take on it would be to make current workers fail so their jobs gets re-queued. But maybe that's already the default behaviour on shut down?

[keyan]

Hey @seivan!

My initial take on it would be to make current workers fail so their jobs gets re-queued. But maybe that's already the default behaviour on shut down?

Yeah that is the current behavior. What I wanted to improve was to allow currently executing jobs to complete (whether successful or not) before restarting.

Any progress on this so far?

So rather than trap the Unix signal I was hoping instead to execute a simple script on the server running my Verk application. The script connects to the node and issues a :stop, which (I believe) invokes the GenServer.stop/3 callback on our GenServers.

#!/usr/bin/env elixir

if Node.connect(:my_named_app), do: Node.spawn(:my_named_app, :init, :stop, [])

I'm still trying to workout the best way to "drain" the executing jobs though. As in, don't kill all the GenServers because some are necessary for finishing execution. But still stop issuing dequeue callbacks.

[edgurgel]

Hm maybe the first step is to implement pause for a queue (on an instance level) and then we can pause all queues so that we can kill the application?

[tlvenn]

With Erlang 19.3, a release will now do a proper stop when it receives a SIGTERM. So it works properly with kubernetes pod shutdown.

[seivan]

@tlvenn so all we have to do is trap those in supervisors and manage them?

[keyan]

With Erlang 19.3, a release will now do a proper stop when it receives a SIGTERM

@tlvenn, this simply removes the requirement of explicitly running the :stop on the node. We still have the aforementioned issue, which is figuring out how to implement stop() callback such that it has the desired behavior of finishing jobs but not dequeuing new ones.

[tlvenn]

@edgurgel have you ever got around implementing the pausing of queues before shutting down ?

[edgurgel]

@tlvenn, sadly not :(

One way of implementing this is having a process as part of the supervision tree that pauses all the queues during shutdown and waits for them as much as timeout then finally shutdown.

I'm not sure when I will have time tbh but it's something that we really want to have :)