Comments (7)
We're planning on using https://github.com/meskyanichi/hirefire for speakerdeck.com. Not sure if there's something better out there, but that was our plan.
from qu.
doh, didn't mean to close this :P
from qu.
Ah great, I asked for qu support already: https://github.com/meskyanichi/hirefire/issues/36
from qu.
Hey there,
(I'm the author of both http://hirefireapp.com/ and https://github.com/meskyanichi/hirefire) and got a request from @fbjork to add support to them. I'm currently working on adding support to http://hirefireapp.com/ - however, in order to do so I'm going to need an additional method in the redis and mongo backends that returns the number of currently working workers.
Without it, there is a possibility that HireFire might spin down workers that are still busy since Qu.length
returns 0 when a job is locked by a worker, and Qu.backend.workers
always returns the number of workers currently running, but aren't necessarily working.
Is it possible to get a method that returns the current amount of workers that are processing a job? Something like:
Qu.working
Qu.workers # this one is currently undefined, unlike Qu.backend.workers
Qu.workers.working
Qu.processing
# etc
With that I could simply do something like this: Qu.length + Qu.working = the amount of jobs in queue or still being processed
, so when this returns > 0
HireFire would not spin down workers. Unlike with Qu.length + Qu.backend.workers
which will always return >= 1
- this would always be inaccurate.
Let me know what you think!
Also, I really like this gem! Great work! I'll probably be using this instead of Resque and Delayed job. The re-scheduling of jobs when a worker is killed (should be excellent on Heroku + HireFire). Also it has a very simple, light-weight code-base and uses low-level Ruby drivers rather than ORM/ODM's which is very appealing as it makes it ODM/ORM independent and easier I assume it's also easier to maintain.
from qu.
@meskyanichi Awesome, thanks for getting in touch. I'm working the web frontend this week, which also needs to know which workers are currently working. I'll try to get something pushed tomorrow for you to use.
Also, I've got a branch adding support for hooks that will probably make life easier for hirefire. I've still got some work to do on it, but you might want to check it out to see the changes.
Thanks for working on this. I was planning on forking hirefire and adding it myself, so I'm glad I don't have to do it and it'll just work!
I'm glad you like Qu so far. It's young, but I think it has a lot of potential!
from qu.
Great. For http://hirefireapp.com/ I don't need to use hooks, since the server constantly checks your queue depth automatically. So for that I only need something like Qu.working
. Once I have that I can pretty much integrate it in a few minutes since it's very simple to add here in this middleware file: https://github.com/meskyanichi/hirefireapp/blob/develop/lib/hirefireapp/middleware.rb and Qu will work.
The other HireFire project over at https://github.com/meskyanichi/hirefire will indeed need hooks. I haven't maintained it in quite a while since I released the hosted service, and have only been pulling in pull requests (the project isn't really stable due to Heroku's platform constraints and invoking hooks (API calls) after a job gets enqueued kind of slows down the HTTP throughput, which, for short jobs, kind of defeats the purpose of having a worker).
Though, if I find the time to dig in to the code again I will. Currently being hammered by feature requests for another open source project. :D But, feel free to fork it and add support for Qu with the hook-based HireFire and I'll pull it in if you need it before I get to adding it.
from qu.
Closing this as it is has aged. We can come back to it later if necessary.
from qu.
Related Issues (20)
- RabbitMQ vs Bunny HOT 3
- Docs don't match current release version? HOT 3
- jobs processed even when queue is not running HOT 6
- [HireFire] Extract a "working" worker's job's queue name HOT 3
- Question: How to handle completed jobs? Any callback? HOT 1
- Auto Redic/Mongo mount while rake spec HOT 1
- Failing test for Mongo backend HOT 1
- db_name must be a string or symbol HOT 6
- Daemonize qu:work HOT 1
- Current status of project HOT 6
- Backend should be notified of failures HOT 1
- Raise error if no queue given to worker HOT 1
- Jobs are currently retried until they succeed
- Worker heartbeat HOT 4
- New gem release? HOT 1
- Move kestrel thrift stuff to its own gem HOT 1
- Release new version HOT 1
- Process payloads in batches HOT 6
- Allow backends to implement batch pop
- Allow backends to implement batch push HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qu.