Comments (15)
Ok so Verk 1.1.0
has support to pausing/resuming queues! 🎉
Now we just need to work on how to pause queues before shutting down (which should be the easiest part :D)
from verk.
Update:
My plan is:
- Release Verk 1.0.0 with the current master
- Release 1.1.0 with pause and resume #127 + Verk Web release with the interface to pause/resume queues
- Work on a simple way to pause queues when SIGTERM/SIGINT is issued (Erlang 19.3+)
from verk.
Thanks for getting back. Yeah, I'm happy to poke around some more and see if I can think of something that might work.
Also, seeing as the answer to my question was "no", I updated the issue title to reflect the new goal.
from verk.
I suppose we just need to implement terminate
on our managers (workers, scheduler, queue)? I will play with erlang 19.3 later this week. Thanks for the new information!
from verk.
Yeah, I think that would be a nice design.
Each worker can conform to two protocols (or we call them Behaviour for elixir I guess)
One for doing actual work and one for doing "cancel"
That way each worker gets granular control on how to handle their cancel. Some might just ignore it, others will handle cancel based on what state they're in.
from verk.
I would love to have this feature.
Yeah it would be nice if we could stop the "managers" queue/schedule etc and avoid any changes on the queues.
Do you want to give it a try?
from verk.
Any progress on this so far?
from verk.
@edgurgel Just quickly glancing this morning, I'd guess https://github.com/edgurgel/verk/blob/master/lib/verk/queue_supervisor.ex would be a place to trap Unix signals and gracefully manage workers.
My initial take on it would be to make current workers fail so their jobs gets re-queued. But maybe that's already the default behaviour on shut down?
from verk.
Hey @seivan!
My initial take on it would be to make current workers fail so their jobs gets re-queued. But maybe that's already the default behaviour on shut down?
Yeah that is the current behavior. What I wanted to improve was to allow currently executing jobs to complete (whether successful or not) before restarting.
Any progress on this so far?
So rather than trap the Unix signal I was hoping instead to execute a simple script on the server running my Verk application. The script connects to the node and issues a :stop
, which (I believe) invokes the GenServer.stop/3 callback on our GenServers.
#!/usr/bin/env elixir
if Node.connect(:my_named_app), do: Node.spawn(:my_named_app, :init, :stop, [])
I'm still trying to workout the best way to "drain" the executing jobs though. As in, don't kill all the GenServers because some are necessary for finishing execution. But still stop issuing dequeue callbacks.
from verk.
Hm maybe the first step is to implement pause for a queue (on an instance level) and then we can pause all queues so that we can kill the application?
from verk.
With Erlang 19.3, a release will now do a proper stop when it receives a SIGTERM. So it works properly with kubernetes pod shutdown.
from verk.
@tlvenn so all we have to do is trap those in supervisors and manage them?
from verk.
With Erlang 19.3, a release will now do a proper stop when it receives a SIGTERM
@tlvenn, this simply removes the requirement of explicitly running the :stop
on the node. We still have the aforementioned issue, which is figuring out how to implement stop()
callback such that it has the desired behavior of finishing jobs but not dequeuing new ones.
from verk.
@edgurgel have you ever got around implementing the pausing of queues before shutting down ?
from verk.
@tlvenn, sadly not :(
One way of implementing this is having a process as part of the supervision tree that pauses all the queues during shutdown and waits for them as much as timeout
then finally shutdown.
I'm not sure when I will have time tbh but it's something that we really want to have :)
from verk.
Related Issues (20)
- Strange behaviour with integer argument, potential bug HOT 17
- Add mix task to reset queues HOT 9
- Add pending jobs to queue stats HOT 2
- Unable to overwrite `max_retry_count` via config.exs HOT 2
- Event for dead job HOT 2
- Question. Can Verk check more frequently for new jobs? HOT 2
- Undocumented `max_dead_jobs` HOT 1
- function Verk.Supervisor.start_link/1 is undefined or private HOT 3
- Im surely missing something. HOT 4
- Coupling of producers and consumers? HOT 1
- Upgrade GenStage to current version HOT 5
- Version Conflict with Confex 3.4.0 HOT 2
- Loosen dependency on redis HOT 2
- Verk 2.0 HOT 12
- Skip queuing jobs during tests. HOT 1
- Batch processing? HOT 2
- Defining contextual data for worker processes HOT 4
- [question] What are the risks of the "experimental" generate_node_id feature? HOT 4
- How do I setup Verk as a producer only? HOT 1
- Connect to Redix via ipv6? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from verk.