bkeepers / qu Goto Github PK

View Code? Open in Web Editor NEW

507.0 9.0 50.0 964 KB

a Ruby library for queuing and processing background jobs.

License: MIT License

Ruby 98.96% Shell 1.04%

qu's Issues

Create a web interface

Should allow viewing workers, inspecting queues, and viewing and trying failures.

Retry logic

How is the retry logic meant to work in qu? If an exception is raised, will the the worker retry the job (5 times be default in mongo) and if it fails write it to the failed collection?

Backend should be notified of failures

Right now backends are not notified when a payload is performed and an exception happens (only failure backend is). The normal backend needs to be notified as well in order for reliable reads to work.

Make poll timer configurable for inferior backends

Some backends will need to poll instead of being able to block waiting for new jobs. That timer should be configurable.

Create an YAML backend

Same of SQL, I think in some small scenarios would be great a builtin YAML or file store backend. So in that way we don't need to have external dependencies or any kind of persistence adapters.

Either include qu-immediate in core or release to rubygems.org

It looks like this was discussed in #36, but it doesn't look like a decision was made. Thanks.

Add deployment information to the wiki

running rake task is simple for development, but what if we updated worker code and want to deploy a new one? somthing should find previous worker pid, kill it, run another. looks like some kind of daemonization needed.

I'm sure we'll figure that out, but we won't be the first who did, so can you please add a section to the README?

Raise error if no queue given to worker

Scheduled jobs ?

Hello,

I would like to know if job scheduling was in the todo list.
It's a feature I often need as a webgame developer! :)

Thanks and good luck!

Allow processing jobs immediately in test mode

Instead of enqueueing a job, it should just be performed immediately.

Create railtie to setup logging and rake tasks

Get Notified By Job Successfully Complete

Dears,

kindly, is there a way to get notified when the Job Done Successfully Completed.
also, how can use qu to add a progress bar to the form for a Job.
something like speakerdeck.com presentation Uploader Processing Progress.

Best Regards,
Shenouda Bertel

Question: How to handle completed jobs? Any callback?

How can I track completed jobs, like: 5 of 10 jobs has been completed?

Loading Rails.logger in qu.rb causes exception

On Rails 2.1 (yes I know...) using:

config.gem 'qu'

Causes rails to bomb out with the exception/error

 Anonymous modules have no name to be referenced by

which is down to qu.rb accessing Rails.logger before it's initialized. This is pretty edge case but I just thought I'd flag an issue for other people that might run into it. You can work round this byrequire 'qu' in boot.rb before loading rails and then configuring the logger yourself.

Error when enqueueing a job: Database names cannot contain the character '.'

I'm using the following versions of things:

rails 3.1
ruby 1.9.2 p290
mongoid 2.2.1
qu 0.1.1
mongo 1.4.0
bson_ext 1.4.0
MongoDB 2.0.0

I am configuring Qu's connection to MongoDB with this bit of code defined in a Rails initializer:

Qu.configure do |c|
    uri = 'mongodb://username:password@localhost:50000/qu_dev'
    c.connection = Mongo::Connection.from_uri uri
end

When I try to enqueue a job I'm getting the following error:

database names cannot contain the character '.'
<gems_path>/mongo-1.4.0/lib/mongo/util/support.rb:54:in `block in validate_db_name'
<gems_path>/mongo-1.4.0/lib/mongo/util/support.rb:52: `each'
<gems_path>/mongo-1.4.0/lib/mongo/util/support.rb:52: `validate_db_name'
<gems_path>/mongo-1.4.0/lib/mongo/db.rb:80:in `initialize'
<gems_path>/mongo-1.4.0/lib/mongo/connectino.rb:312:in `new'
<gems_path>/mongo-1.4.0/lib/mongo/connectino.rb:312:in `[]'
<gems_path>/qu-mongo-0.1.1/lib/qu/backend/mongo.rb:122:in `[]'
<gems_path>/qu-mongo-0.1.1/lib/qu/backend/mongo.rb:118:in `jobs'
<gems_path>/qu-mongo-0.1.1/lib/qu/backend/mongo.rb:42:in `enqueue' 
<gems_path>/qu-0.1.1/lib/qu.rb:29:in `enqueue'

The name of my job was "tasks":

class TaskJob
  @queue = :tasks

   self.perform
   end

This problem also occurred when using the Mongo Ruby driver v1.3.1.

Looking at the code it appears that qu/backend/mongo.rb#[ ] method is trying to get a reference to a collection but it's using the Mongo::Connection#[ ] method to do it, which actually gives back a Mongo::DB instance. (the API for version 1.3.1 of the Mongo Ruby driver also behaves the same way as 1.4.0).

I suppose I'm doing something wrong since enqueueing is so fundamental to Qu :)

(hooks) Job initialize called on create

Hi.

In hooks branch you suggest to move all finders into job initialize method, and only this method will get arguments.

But you call initialize on job creation, so this will double my DB queries. Yes I can do some thing like this:


class ProcessPresentation < Qu::Job
  def initialize(presentation_id)
    @presentation_id = presentation_id
  end

  def perform
     Presentation.find(@presentation_id).process!
  end
end

But this looks like a hack for me.

Worker heartbeat

I'd love optional heartbeat tracking. NullHeartbeat by default, but allow people to inject a redis or whatever based heartbeat that is similar to https://github.com/shaiguitar/resque_stuck_queue. Something that gives people insight into the workers.

Optionally fork before running each job

This should be an option that is easy to turn on or off.

New Relic RPM instrumentation

Would be great if someone could add Qu instrumentation to the rpm_contrib gem to get valuable metrics from the Qu workers.

findAndModify returns nil in Mongo 2.0

The mongo backend will raise an error on Mongo 2 due to the changing of #find_and_modify to return nil instead of raising an error.

Question: How can I set a task to be performed only when it reaches an specific date?

I want to create a Task, that contains the day it which it was created. When the task is, say, 12h old it should perform its task.

What could be an elegant way to do this?

db_name must be a string or symbol

I'm not sure why I'm getting this now, but it seems to have just started happening. My app is running on Heroku and my app connects just fine to MongoHQ using this: MongoMapper.config = { Rails.env => { 'uri' => ENV['MONGOHQ_URL'] } }. However, whenever I try to create a Qu job (or even check the length), it throws this error:

pry(main)>Qu.length
TypeError: db_name must be a string or symbol
from /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/support.rb:49:in `validate_db_name'

My config/qu.rb looks like this:

Qu.configure do |c|
  # if ENV['MONGOHQ_URL']
  #   c.connection  = Mongo::Connection.from_uri(ENV['MONGOHQ_URL'])
  # else
  #   c.connection  = Mongo::Connection.new('localhost', 27017).db("action-log-#{Rails.env}")
  # end
  # if Rails.env == "staging"
  #   c.connection = Mongo::Connection.from_uri("--uri removed--")
  # end
  c.logger = Logger.new('log/qu.log')
end

I can't seem to figure out what is causing this. It would seem that the db_name is not getting set properly and is probably nil, causing the validation error to fire. But this configuration used to work just fine.

Any help would be appreciated. Thanks.

Update: added a backtrace

[5] pry(main)> Qu.length
TypeError: db_name must be a string or symbol
from /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/support.rb:49:in `validate_db_name'
[6] pry(main)> wtf?
Exception: TypeError: db_name must be a string or symbol
--
0: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/support.rb:49:in `validate_db_name'
1: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/db.rb:91:in `initialize'
2: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `new'
3: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `[]'
4: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:244:in `block in apply_saved_authentication'
5: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `each'
6: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `apply_saved_authentication'
7: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:174:in `checkout_new_socket'
8: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:272:in `block (2 levels) in checkout'
9: <internal:prelude>:10:in `synchronize'
[7] pry(main)>

Update: better backtrace

2013-06-10T18:36:57.072430+00:00 app[web.1]: Started GET "/hirefire/hirefiretoken/info" for 99.109.254.3 at 2013-06-10 14:36:57 -0400
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/support.rb:49:in `validate_db_name'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/db.rb:91:in `initialize'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `new'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `[]'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:244:in `block in apply_saved_authentication'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `each'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `apply_saved_authentication'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:174:in `checkout_new_socket'
2013-06-10T18:36:57.083486+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:272:in `block (2 levels) in checkout'
2013-06-10T18:36:57.083486+00:00 app[web.1]: <internal:prelude>:10:in `synchronize'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:263:in `block in checkout'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:256:in `loop'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:256:in `checkout'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:530:in `checkout_reader'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:541:in `checkout_socket_from_connection'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:469:in `block in send_initial_query'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:33:in `block in instrument'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:65:in `instrument'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:32:in `instrument'
2013-06-10T18:36:57.083659+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:466:in `send_initial_query'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:451:in `refresh'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:110:in `next'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/db.rb:527:in `command'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:173:in `count'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/collection.rb:963:in `count'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/qu-mongo-0.2.0/lib/qu/backend/mongo.rb:55:in `length'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/config/initializers/hirefire.rb:4:in `block (2 levels) in <top (required)>'
2013-06-10T18:36:57.066828+00:00 heroku[router]: at=info method=GET path=/hirefire/hirefiretoken/info host=blooming-rain-6086.herokuapp.com fwd="99.109.254.3" dyno=web.1 connect=2ms service=22ms status=200 bytes=35
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:53:in `call'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:53:in `block in dynos'
2013-06-10T18:36:57.083811+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `each'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `inject'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `dynos'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:42:in `each'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:58:in `digest_body'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:26:in `call'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/head.rb:14:in `call'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/flash.rb:242:in `call'
2013-06-10T18:36:57.084226+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/cookies.rb:341:in `call'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `_run__2616270529350534789__call__644675313378680830__callbacks'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
2013-06-10T18:36:57.084380+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:32:in `call_app'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `block in call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/tagged_logging.rb:22:in `tagged'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/request_id.rb:22:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
2013-06-10T18:36:57.084695+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/static.rb:63:in `call'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-ssl-1.3.3/lib/rack/ssl.rb:27:in `call'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-timeout-0.0.4/lib/rack/timeout.rb:16:in `block in call'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/ruby-1.9.3/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
2013-06-10T18:36:57.084871+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-timeout-0.0.4/lib/rack/timeout.rb:16:in `call'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/engine.rb:479:in `call'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:223:in `call'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/content_length.rb:14:in `call'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/log_tailer.rb:17:in `call'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:81:in `block in pre_process'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `catch'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `pre_process'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:54:in `process'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:39:in `receive_data'
2013-06-10T18:36:57.085210+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/backends/base.rb:63:in `start'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/server.rb:159:in `start'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/handler/thin.rb:13:in `run'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/server.rb:268:in `start'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/server.rb:70:in `start'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:55:in `block in <top (required)>'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:50:in `tap'
2013-06-10T18:36:57.085360+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:50:in `<top (required)>'
2013-06-10T18:36:57.085360+00:00 app[web.1]: script/rails:6:in `require'
2013-06-10T18:36:57.085715+00:00 app[web.1]: script/rails:6:in `<main>'
2013-06-10T18:37:17.867331+00:00 app[web.1]: Started GET "/hirefire/hirefiretoken/info" for 184.72.165.101 at 2013-06-10 14:37:17 -0400
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/support.rb:49:in `validate_db_name'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/db.rb:91:in `initialize'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `new'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:348:in `[]'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:244:in `block in apply_saved_authentication'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `each'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:243:in `apply_saved_authentication'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:174:in `checkout_new_socket'
2013-06-10T18:37:17.870096+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:272:in `block (2 levels) in checkout'
2013-06-10T18:37:17.870096+00:00 app[web.1]: <internal:prelude>:10:in `synchronize'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:263:in `block in checkout'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:256:in `loop'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool.rb:256:in `checkout'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:530:in `checkout_reader'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:541:in `checkout_socket_from_connection'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:469:in `block in send_initial_query'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:33:in `block in instrument'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:65:in `instrument'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/util/logging.rb:32:in `instrument'
2013-06-10T18:37:17.870280+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:466:in `send_initial_query'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:451:in `refresh'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:110:in `next'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/db.rb:527:in `command'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/cursor.rb:173:in `count'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/mongo-1.8.6/lib/mongo/collection.rb:963:in `count'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/qu-mongo-0.2.0/lib/qu/backend/mongo.rb:55:in `length'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/config/initializers/hirefire.rb:4:in `block (2 levels) in <top (required)>'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:53:in `call'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:53:in `block in dynos'
2013-06-10T18:37:17.870434+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `each'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `inject'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:52:in `dynos'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/hirefire-resource-0.1.1/lib/hirefire/middleware.rb:42:in `each'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:58:in `digest_body'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/etag.rb:26:in `call'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/conditionalget.rb:25:in `call'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/head.rb:14:in `call'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/params_parser.rb:21:in `call'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/flash.rb:242:in `call'
2013-06-10T18:37:17.870835+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:210:in `context'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/session/abstract/id.rb:205:in `call'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/cookies.rb:341:in `call'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `_run__2616270529350534789__call__644675313378680830__callbacks'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:405:in `__run_callback'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:385:in `_run_call_callbacks'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/callbacks.rb:81:in `run_callbacks'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/remote_ip.rb:31:in `call'
2013-06-10T18:37:17.870988+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:32:in `call_app'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `block in call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/tagged_logging.rb:22:in `tagged'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/logger.rb:16:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/request_id.rb:22:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/methodoverride.rb:21:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/runtime.rb:17:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.13/lib/active_support/cache/strategy/local_cache.rb:72:in `call'
2013-06-10T18:37:17.871306+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/lock.rb:15:in `call'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.13/lib/action_dispatch/middleware/static.rb:63:in `call'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-ssl-1.3.3/lib/rack/ssl.rb:27:in `call'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-timeout-0.0.4/lib/rack/timeout.rb:16:in `block in call'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/ruby-1.9.3/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
2013-06-10T18:37:17.871458+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-timeout-0.0.4/lib/rack/timeout.rb:16:in `call'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/engine.rb:479:in `call'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/application.rb:223:in `call'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/content_length.rb:14:in `call'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/rack/log_tailer.rb:17:in `call'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:81:in `block in pre_process'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `catch'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:79:in `pre_process'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:54:in `process'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/connection.rb:39:in `receive_data'
2013-06-10T18:37:17.871762+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/backends/base.rb:63:in `start'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.5.1/lib/thin/server.rb:159:in `start'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/handler/thin.rb:13:in `run'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.5/lib/rack/server.rb:268:in `start'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands/server.rb:70:in `start'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:55:in `block in <top (required)>'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:50:in `tap'
2013-06-10T18:37:17.871911+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.13/lib/rails/commands.rb:50:in `<top (required)>'
2013-06-10T18:37:17.871911+00:00 app[web.1]: script/rails:6:in `require'
2013-06-10T18:37:17.872253+00:00 app[web.1]: script/rails:6:in `<main>'

Changelog

Would love to see a changelog in the repo if possible!

Add hooks

:enqueue
:before
:after
:failure
:complete
:requeue
:release

exception in unregister_worker

Hi.

unregister_worker method in Backend classes assume to get id as parameter, but Worker class in start method (line 61) pass worker object to it.

And because of this we get exception:

Cannot serialize an object of class Qu::Worker into BSON.
/Users/gregory/.rvm/gems/ree-1.8.7-2011.03@applicaster-encoder/gems/bson-1.4.0/lib/../lib/bson/bson_c.rb:24:in `serialize'
/Users/gregory/.rvm/gems/ree-1.8.7-2011.03@applicaster-encoder/gems/bson-1.4.0/lib/../lib/bson/bson_c.rb:24:in `serialize'
/Users/gregory/.rvm/gems/ree-1.8.7-2011.03@applicaster-encoder/gems/mongo-1.3.1/lib/../lib/mongo/collection.rb:344:in `remove'
/Users/gregory/.rvm/gems/ree-1.8.7-2011.03@applicaster-encoder/gems/qu-mongo-0.1.0/lib/qu/backend/mongo.rb:87:in `unregister_worker'
/Users/gregory/.rvm/gems/ree-1.8.7-2011.03@applicaster-encoder/gems/qu-0.1.0/lib/qu/worker.rb:49:in `start'
.....

And though the specs are passed, qu raise exception in real projects. Maybe you need any kind of integration tests?

Thanks.

Duplicate collections

Not sure if this is qu related or not, but I thought I'd share it in case. I just updated to the latest version of qu-mongo and deployed to Heroku. All of a sudden all the qu collections are duplicate!

These are the collections in my qu database:
qu.workers
qu.queue:sync_friends
qu.queues
qu.queue:sync_account
qu.workers
qu.queue:sync_friends
qu.queues
qu.queue:sync_account

Any ideas?

Daemonize qu:work

rake qu:work does its job (literally :) but when I close the ssh connection it just stops. Any suggestion to "daemonize" it?

Auto Redic/Mongo mount while rake spec

It'd be just handy to automatically setup Redis and Mongo server instances while trying to run tests.

New gem release?

The last version (0.2.0) of qu has been released 2 years ago. Are there plans to release a new version of the gem?

If I want to use it in production soon, is it recommended to use version 0.2.0, or the latest master branch?

mongodb - use tailable cursors instead of polling

[http://www.captaincodeman.com/2011/05/28/simple-service-bus-message-queue-mongodb/](C# demonstration code)
[https://gist.github.com/665978](Ruby tailable cursor code).

Current status of project

Hi there... I've implemented Qu in a current project and am liking it so far. Looking through the issues here, as well as the Google Group, I get the feeling that this project has stalled in terms of recent updates... is this the case? The questions I have posted to the group don't appear (as if they aren't being moderated), and I seem pull requests and issues from months and years ago not addresses.

Is Qu being supported any longer, or is there another branch of this that is more active?

Thanks,
Adam

Qu is unlicensed

I can't find anywhere where Qu declares a license.

Stats

Would be great if you could add stats to each queue, i.e processed, failed

Create an SQL backend

SQL databases don't make the best queues, but for smaller apps with lower throughput, they work just fine.

It would be nice to have an SQL backend that is as close to the driver as possible, but I'm not aware of any stand-alone libraries that abstract the various database drivers.

Failing test for Mongo backend

The "connection should use MONGOHQ_URL from heroku" test of the Mongo backend is failing due to a change in client gem version 1.8.2 and higher.

I don't use MongoDB so I'm neither sure how to fix this nor particularly interested in doing so.

Jobs are currently retried until they succeed

This is different from qu's currently released behavior which is to simply store failures for perusing at a later time. While most jobs can be retried and will succeed on a retry, there are some that would be retried forever. It would probably be good to allow the backend to decide what to do when a job raises an exception, rather than having the worker abort which will cause a retry.

Invalid gemspec

Got this exception when trying to push to Heroku using the latest gem:

WARNING: #<ArgumentError: Illformed requirement ["#Syck::DefaultKey:0x00000002594708 0.1.1"]>

-- encoding: utf-8 --

Gem::Specification.new do |s|
s.name = %q{qu-redis}
s.version = "0.1.1"

s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
s.authors = ["Brandon Keepers"]
s.date = %q{2011-10-03 00:00:00.000000000Z}
s.description = %q{Redis backend for qu}
s.email = ["[email protected]"]
s.files = ["lib/qu-redis.rb", "lib/qu/backend/redis.rb"]
s.homepage = %q{http://github.com/bkeepers/qu}
s.require_paths = ["lib"]
s.rubygems_version = %q{1.3.7}
s.summary = %q{Redis backend for qu}

if s.respond_to? :specification_version then
current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
s.specification_version = 3

if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
s.add_runtime_dependency(%q, [">= 0"])
s.add_runtime_dependency(%q<simple_uuid>, [">= 0"])
s.add_runtime_dependency(%q, ["#Syck::DefaultKey:0x00000002594708 0.1.1"])
else
s.add_dependency(%q, [">= 0"])
s.add_dependency(%q<simple_uuid>, [">= 0"])
s.add_dependency(%q, ["#Syck::DefaultKey:0x00000002594708 0.1.1"])
end
else
s.add_dependency(%q, [">= 0"])
s.add_dependency(%q<simple_uuid>, [">= 0"])
s.add_dependency(%q, ["#Syck::DefaultKey:0x00000002594708 0.1.1"])
end
end
WARNING: Invalid .gemspec format in 'vendor/bundle/ruby/1.9.1/specifications/qu-redis-0.1.1.gemspec'
Could not find qu-redis-0.1.1 in any of the sources

autoscaling support

Not sure if this is the right forum for this, but it would be great to see functionality added to support autoscaling on Heroku. Perhaps another gem like qu-autoscale or something along those lines?

BSON::InvalidDocument exception

Getting an exception when I run the following:

options = { generated_at: Time.current, meta: { state: :processed, source: 'Chat', client: 'iPhone' } }
Qu.enqueue(StatsWorker, "Push Notifications", 1, options)

Stack trace:
BSON::InvalidDocument exception raised:

/app/vendor/bundle/ruby/1.9.1/gems/bson-1.3.1/lib/bson/bson_c.rb:24:in `serialize'
/app/vendor/bundle/ruby/1.9.1/gems/bson-1.3.1/lib/bson/bson_c.rb:24:in `serialize'
/app/vendor/bundle/ruby/1.9.1/gems/mongo-1.3.1/lib/mongo/collection.rb:872:in `block in insert_documents'
/app/vendor/bundle/ruby/1.9.1/gems/mongo-1.3.1/lib/mongo/collection.rb:871:in `each'
/app/vendor/bundle/ruby/1.9.1/gems/mongo-1.3.1/lib/mongo/collection.rb:871:in `insert_documents'
/app/vendor/bundle/ruby/1.9.1/gems/mongo-1.3.1/lib/mongo/collection.rb:305:in `insert'
/app/vendor/bundle/ruby/1.9.1/gems/qu-mongo-0.1.3/lib/qu/backend/mongo.rb:55:in `enqueue'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu.rb:29:in `enqueue'
/app/app/models/apn/notification.rb:87:in `process_push!'
/app/vendor/bundle/ruby/1.9.1/gems/stateflow-0.4.2/lib/stateflow/transition.rb:33:in `execute_action'
/app/vendor/bundle/ruby/1.9.1/gems/stateflow-0.4.2/lib/stateflow/transition.rb:23:in `find_to_state'
/app/vendor/bundle/ruby/1.9.1/gems/stateflow-0.4.2/lib/stateflow/event.rb:19:in `fire'
/app/vendor/bundle/ruby/1.9.1/gems/stateflow-0.4.2/lib/stateflow.rb:61:in `fire_event'
/app/vendor/bundle/ruby/1.9.1/gems/stateflow-0.4.2/lib/stateflow.rb:35:in `block (2 levels) in stateflow'
/app/app/workers/push_notification_worker.rb:7:in `block in perform'
/app/app/workers/abstract_worker.rb:8:in `block (2 levels) in wrap'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.2.3/lib/mongoid.rb:130:in `unit_of_work'
/app/app/workers/abstract_worker.rb:8:in `block in wrap'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.1.1/lib/active_support/benchmarkable.rb:43:in `block in benchmark'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.1.1/lib/active_support/core_ext/benchmark.rb:5:in `block in ms'
/usr/local/lib/ruby/1.9.1/benchmark.rb:309:in `realtime'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.1.1/lib/active_support/core_ext/benchmark.rb:5:in `ms'
/app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.1.1/lib/active_support/benchmarkable.rb:43:in `benchmark'
/app/app/workers/abstract_worker.rb:7:in `wrap'
/app/app/workers/push_notification_worker.rb:5:in `perform'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/payload.rb:23:in `perform'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/worker.rb:49:in `work'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/worker.rb:57:in `block in start'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/worker.rb:57:in `loop'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/worker.rb:57:in `start'
/app/vendor/bundle/ruby/1.9.1/gems/qu-0.1.3/lib/qu/tasks.rb:5:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/task.rb:205:in `call'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:112:in `invoke_task'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:90:in `block (2 levels) in top_level'
/app/vendor/bundle/ruby/1.9.1/gems/rake-0.9.2/lib/rake/application.rb:62:in `block in run'
/app/vendor/bundle/ruby/1.9.1/bin/rake:19:in `load'

[HireFire] Extract a "working" worker's job's queue name

Hey guys,

I am currently in the process of finishing up a full rewrite of the HireFire system, and I am currently trying to add support for the Qu worker library as well as it's been requested by a bunch of HireFire users.

Thus far it seems to be working pretty well, but I am still missing something, namely the ability to see which job a worker is currently processing (if any). Is it possible to extract that data with the current code base?

Here is an implementation for counting queues in Resque and Qu for the new HireFire client gem.

https://gist.github.com/15962d22d41562e03323

Notice how I commented out two parts in the Qu macro. These are the things I am unsure of how to do. However, it might not even be necessary but I'd like to confirm this with you.

The scenario is as follows; When you pop a job from the queue in for example Redis, and immediately call Qu.backend.length, will it return 0 or 1? And if it returns 0 could that mean the worker is still actually processing the job or has the worker definitely finished processing the job? (Assuming there was only 1 job in the queue before pop-ing).

Also, does this result differ from the Mongo backend?

HireFire is basically an autoscaler for web and worker dynos on Heroku. Now, it might be a problem if a worker is still processing a job, but Qu is reporting 0 so HireFire will think it's clear to spin down all workers, potentially causing the worker to be interrupted and fails the job. Resque provides a method to return all the workers that are currently "working", and they contain the job object and their data (such as the name of the queue the job was pushed in to). Using that information I am capable of doing a total_queue_size + total_workers_working to ensure that when 0 is returned, it means there is definitely nothing in the queue, and there is nothing being processed from the queue by any worker.

Any help/tips/feedback much appreciated!

Cheers,
Michael

Add ability for a job to timeout

If a job is scheduled that is taking a long time or is hung up for some reason, it would be nice if Qu could stop/kill the job and then let some hook/callback be invoked so we could know it happened. At that point we could log a message, raise an exception, requeue the job, etc.

Make exception handling configurable

Resque does a nice job of this by allowing you to swap exception handling backends.

Infinite loop when reserving jobs

When using the latest qu-mongo the following message keeps repeating in my development.log

Reserving job in queue sync_friends
Reserving job in queue sync_friends
Reserving job in queue sync_friends
Reserving job in queue sync_friends
Reserving job in queue sync_friends

Any idea what's going?

Redis jobs not being cleaned out properly

Hi,

I'm using the latest version of Qu. When I check each queue length it says zero. However when I list keys * in Redis I get this: https://gist.github.com/6ed74e94daccf4cfa0ff

Any ideas?

This is on Heroku.

Release new version

There have been a ton of breaking changes. Time to look over everything and release a new usable version.

Invalid use of `retry`

When I run qu-mongo with ruby 1.9.2p290, it appears the use of retry is invalid.

Specifically the error from ruby is just "Invalid retry".

It seems as though the context in which you are using retry would have worked in Ruby 1.8 but no longer does in 1.9. I believe the idea of retry is to do something again if an exception occurs.

reference: http://www.ruby-forum.com/topic/184694

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble