utgarda / sidekiq-status Goto Github PK

Hi,

I am running sidekiq status in inline mode, since I want to execute the worker right away.
the worker runs fine, but the status in redis remains in queue forever.

in development.rb I have.
require 'sidekiq/testing'
Sidekiq::Testing.inline!

So the question is how can I get the status when running inline?

The version of Rubygems still has sidekiq (< 3.1, >= 2.7), whereas the one on github has the Sidekiq requirements relaxed.

Hello,

Wouldn't it make more sense to have this kind of interface:

class MyJob
  include Sidekiq::Worker
  include Sidekiq::Status::Worker # Important!

  def perform(*args)
    total 100 # by default
    10.times do |i|
      at 10 * i, "message #{i}"
      sleep 1
    end
    sleep 5
    at 100, "done"
  end
end

Also, Sidekiq::Status::num should be named Sidekiq::Status::at imho.

I can create a pull request if you want.

In light of you accepting the pull request #12, it'd be great if you released a new minor version (0.3.2), so that we wouldn't be forced to grab the master in bundler.
Please let me know if you want me to do it.

hi all,
I am using delayed job extension for sidekiq and want to use the sidekiq status gem to know status of each job

Sidekiq has processed all the jobs but the status is showing queued only .Please tell whether the gem is compatible with delayed job since now worker/jid is coming in the view.
Redis 3.2
rails 3.2.21
sidekiq 4.2.1

• Need to update Sidekiq::Status::Storage#read_field_for_id, this method using hmget command which returns array and method always returning first value from return value instead method can you hget command which returns value for given field so updated method will be like

   def read_field_for_id(id, field)
    Sidekiq.redis do |conn|
      conn.hget(key(id), field)
     end
  end

• Also Sidekiq::Status::Web need to refactor current code is bit complex.

Say I compile an array of hashes in the worker, and set them to be retrieved using a key like this:

@array_of_hashes = [{"key" => "value"}, {"key2" => "value2"}]
store keys_and_values: @array_of_hashes.to_s
keys_and_values = retrieve :keys_and_values

I can now access keys_and_values in a controller like this:

@data = Sidekiq::Status::get_all job_id # job_id is the ID of the job which has been stored somewhere

And render it to the browser as JSON like this:

render json: @data

The data comes out groovily, except for all the horrible key-value stuff, which comes out like this:

keys_and_values: "[{"key" => "value"}, {"key2" => "value2"}]"

And I can't parse that to valid JSON because of all the quotations.

But if I try to return array data without calling to_s on it first, I'm not returning strings and Sidekiq can't handle it.

How oh how could I resolve this problem?

If I reboot my worker when it runs Sidekiq::Status.status always returns :working, although the job is not running anymore. Is there a way to figure out that my job was interrupted?

I noticed on Travis CI, Build is failing for ruby 2.1 & 2.2 getting following error

NoMethodError: undefined method 'spec' for nil:NilClass
An error occurred while installing
sidekiq-status (0.5.3), and Bundler
cannot continue.
Make sure that gem install sidekiq-status -v '0.5.3' succeeds
before bundling.

Seems this issue is specific to Travis CI config/settings

@utgarda Can you look into it.

Fix current master build as
require 'celluloid' replaced with require 'celluloid/current' (supported in celluloid ~> 0.17) to avoid BACKPORTED warning.
But sidekiq '2.7' using celluloid '0.12.0' in which require 'celluloid/current' not supported.

Hi,

I try to run my rspec but i got this error

Redis::CannotConnectError:
       Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)

Here are my sidekiq.rb

require 'sidekiq'
require 'sidekiq-status'

Sidekiq.configure_server do |config|
  config.redis = { :url => "redis://#{CONFIG[:redis_server]}:#{CONFIG[:redis_port]}"}
  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
  end
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
end

Sidekiq.configure_client do |config|
  config.redis = { :url => "redis://#{CONFIG[:redis_server]}:#{CONFIG[:redis_port]}"}
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
end

If i remove the middleware all my tests green.
How to stub redis inside sidekiq-status? Can you give me an example to stub it or to solve this? I'm using fakeredis also.
My goal is i want my test to be isolated (Not connecting to other services such as Redis)

It looks like there is no way to set expiration when queueing a worker

def call(worker_class, msg, queue, redis_pool=nil)
      store_status msg['jid'], :queued, nil, redis_pool
      yield
end

Third parameter is nil and default expiration Sidekiq::Status::DEFAULT_EXPIRY will be used.

My use case is: I use one worker that creates several K of subworkers and then waits for them to complete with this code:

WORKING_STATUS = [:queued, :working, :failed]
jobs = []
...
smth.each do
  jobs << CustomerImporter.perform_async(connector.id, customer.id)
  ...
end
while jobs.size > 0
  jobs = jobs.select{|job_id| WORKING_STATUS.include? Sidekiq::Status::status(job_id)}
  sleep 10
end

About an hour the main worker stops (with several K subworkers in the queue).
Expiration set to 5h:

chain.add Sidekiq::Status::ServerMiddleware, expiration: 300.minutes

I couldn't find on the docs what does passing expiration option to middleware does.

Sidekiq::status seems to never get updated when running Sidekiq in inline mode in test environment.

Part of my rails_helper.rb:

config.around(:each) do |example|
  Sidekiq::Testing.inline!(&example)
end

This can be remedied by store status: 'complete', but it's hacky. The best solution would be to avoid Redis when testing.

If there's interest I might take a stab at implementing this.

queue = Sidekiq::Queue.new
    queue.each do |job|
      if job.jid ==  session[:job_id]
         job_removed=job.delete

      end
    end

Even after deleting the job from the queue, running ap Sidekiq::Status::status(session[:job_id]) keeps giving me :queued. I have to run Sidekiq.redis { |conn| conn.flushdb } to clear all the jobs from the redis. Any way to solve this issue?

http://cl.ly/image/0L0H0w2y1o3x

Basically I delete the item from a queue within the UI, and then after I queue the status it is still listed as :queued

[6] pry(main)> status = Sidekiq::Status::status('4f5d19bcf07013ecfbc38486')
=> :queued

Hi, I'm using sidekiq status 0.5.0 with sidekiq 3.0.2, after running 2 jobs this is what I see on rdm:

however the sidekiq statuses tab is empty, I was first using sidekiq with set namespace: 'sidekiq' on the redis connection, I've then removed that but still no statuses can be seen.

Any thoughts?

Sidekiq 3.5 was released a a few weeks ago, but we can't upgrade as sidekiq-status is locked to 3.3.*

I setup everything according to readme. However, once i open "Statuses" tab in WebUI, I get:

09:25:09 web.1    | SyntaxError - /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:44: syntax error, unexpected ';'
09:25:09 web.1    | ...statuses.each do |container| -; @_out_buf.concat "\n"
09:25:09 web.1    | ...                               ^
09:25:09 web.1    | /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:62: syntax error, unexpected ';'
09:25:09 web.1    | ; @_out_buf.concat "  ";  end -; @_out_buf.concat "\n"
09:25:09 web.1    |                                 ^
09:25:09 web.1    | /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:63: syntax error, unexpected ';'
09:25:09 web.1    | ; @_out_buf.concat "  ";  if @statuses.empty? -; @_out_buf.concat "\n"
09:25:09 web.1    |                                                 ^
09:25:09 web.1    | /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:67: syntax error, unexpected ';'
09:25:09 web.1    | ; @_out_buf.concat "  ";  end -; @_out_buf.concat "\n"
09:25:09 web.1    |                                 ^
09:25:09 web.1    | /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:70: syntax error, unexpected keyword_ensure, expecting keyword_end
09:25:09 web.1    | /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:74: syntax error, unexpected keyword_end, expecting end-of-input
09:25:09 web.1    | end;end;end;end
09:25:09 web.1    |                ^:
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/tilt-1.4.1/lib/tilt/template.rb:264:in `class_eval'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/tilt-1.4.1/lib/tilt/template.rb:264:in `compile_template_method'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/tilt-1.4.1/lib/tilt/template.rb:240:in `compiled_method'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/tilt-1.4.1/lib/tilt/template.rb:169:in `evaluate'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/tilt-1.4.1/lib/tilt/template.rb:103:in `render'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:805:in `render'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:664:in `erb'
09:25:09 web.1    |     /home/vagrant/.rvm/gems/ruby-2.1.0/gems/sidekiq-status-0.4.0/lib/sidekiq-status/web.rb:33:in `block in registered'

From the code, I understand that only scheduled jobs can be unscheduled..

Is there a way to cancel/delete enqueued jobs that are yet to be push to workers?

Hello,

This gem is very useful for us. Thanks for developing it.

I am facing a problem that arguments format in normal sidekiq listing and status listing is totally different. For status listing arguments is not in readable format. Check the images below:

First image is listing of your plugin. Second image is listing of standard sidekiq. You will notice how easy and readable is the standard sidekiq listing. In status listing we can't ready which job with which arguments was run.

Please suggest how to fix it.

Currently there is no way to overwrite sidekiq globally set in Sidekiq::Status::ServerMiddleware definition :expiration value. It's used globally for all workers. In my case I need a worker with custom expiration time different from other workers. It's not doable for now. I have an idea how to do it and I will propose a pull request with it.

There are also some weird parts of code regarding expiration:

• https://github.com/utgarda/sidekiq-status/blob/master/lib/sidekiq-status/server_middleware.rb#L27 - because it checks method :expiration but uses worker.expiration= ...

In this pull request I will also make it work better.

Currently specs for Sidekiq::Status::Web are failing.

Added TODO for fixing this in my PR #67, problem is Sidekiq::Workers.new API returning nil queue which is the cause of failing specs.

This can fixed along #44, am working this currently.

I'm not seeing any status (or other data) for jobs created using this kind of call:

job_id = MyWorker.perform_async(arg1, arg2) # this successfully generates a job_id
# time passes, some work is done...
pct_complete = Sidekiq::Status::pct_complete job_id # always Nan
completed = Sidekiq::Status::complete? job_id # always false

Here's my worker, which does it's work successfully:

class MyWorker
  include Sidekiq::Worker
  include Sidekiq::Status::Worker

  def perform(arg1, arg2)
    begin
      array_of_people = magically_get_array_of_people # ...

      total array_of_people.size
      at 0, "Starting..."

      array_of_people.each_with_index do |f, index|
        at index, "Working..."
      end
    rescue => ex
      logger.warn ex
    end
  end
end

Sidekiq::Status always returns Nan, nil, etc. for all doc'd commands. The gem doesn't appear to actually receiving the updates generated in the Worker. Seems like it might be a config issue... so here's my config/initializers/Sidekiq.rb:

require 'sidekiq'
require 'sidekiq-status'

Sidekiq.configure_client do |config|
  config.redis = { :url => ENV["REDISTOGO_URL"], :size => 1 }
  # sidekiq-status
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
  #
end

Sidekiq.configure_server do |config|
  config.redis = { :url => ENV["REDISTOGO_URL"], :size => 2 }
  # sidekiq-status
  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
  end
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end
  #
end

Any hints on a likely culprit here?

Okay, this is what I'm doing...

I have a form that uploads a csv file and creates a db record.
It stores the job_id in the record so that we can reference it's status on the index screen in a list of the jobs.

Whenever I call Sidekiq::Status::get_all(record.job_id) it just returns nil
Same goes for Sidekiq::Status::pct_complete(record.job_id), etc.

Any reason for this?

.../vendor/bundle/ruby/2.2.0/gems/sidekiq-3.4.1/lib/sidekiq.rb:103:in `server_middleware': uninitialized constant Sidekiq::Processor (NameError)

bin/workers

require 'sidekiq'
require 'sidekiq-status'

...

Sidekiq.configure_client do |cfg|
  cfg.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end

 ...
end

Sidekiq.configure_server do |cfg|

  #### Only this part is causing the problem:
  cfg.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 10.minutes
  end

  cfg.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware
  end

  ...

end

rails initializer

  require 'sidekiq-status'

  # 设定 sidekiq-status 中间件.
  Sidekiq.configure_client do |config|
    config.client_middleware do |chain|
      chain.add Sidekiq::Status::ClientMiddleware
    end
  end

  Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
      chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
    end
    config.client_middleware do |chain|
      chain.add Sidekiq::Status::ClientMiddleware
    end
  end

job define:

class Order::RemitJob < ActiveJob::Base
  include Sidekiq::Worker

  def perform(order)
    order.transition_to! 'finished'
  end
end

model define

  def remit_job
    job = RemitJob.set(wait: Rails.application.config.x.buyer_auto_receive_delay.hours).perform_later(self)
    status = Sidekiq::Status::status(job.job_id)   # => here always return nil.
  end

I can confirm Sidekiq is started and worked for me, But I can't get it status with this gem.

It'd be great if you could specify which queue to check for the job id.

Eg Sidekiq::Status.working?(job_id, :queue => "high")

I'm moving from resque to sidekiq, and noticed that with sidekiq-status I can't see the information for completed jobs as I could with resque-status.
So the jobs appear only on the 'Statuses' list when they are being processed, but disappear once they complete, no matter the expiration.
Is this normal behavior or am I doing something wrong?

According to the docs; "Since inlining your sidekiq worker will run it in-process, any exception it throws will make your test fail." Am I missing any obvious reasons for why it has to be so?
This is making it hard to test; the interface for perform_async method is changed depending on if we're testing or not. Also, how do you get the job id if perform raises an exception?

Rubygems shows v0.6.0 was released on December 27, 2015.

And the diff looks non-malicious. It pretty much corresponds with this: v0.5.4...d92fca4

So it looks legitimate (making this a very minor issue). But it could use a v0.6.0 tag.

I am using Sidekiq::Status inside an app that relies heavily on the status reported by this gem, esp. since the status is displayed to the users of this application.

I found the current way of testing a bit brittle, since all it does is to mock the return status as :complete. I understand the fact that inline way of testing sidekiq will always give :complete status. But, Sidekiq also has Sidekiq::Testing.fake!, which when used should return status as per the job's actual status. That is, I should be able to do this:

RSpec.describe MyWorker do
  it 'returns the current status of the job' do
    jid = described_class.perform_async(user.id, echo_url)
    expect(Sidekiq::Status.status(jid)).to eq(:queued)
    described_class.drain
    expect(Sidekiq::Status.status(jid)).to eq(:complete)
  end
end

However, since Sidekiq does not run Server Middlewares with such tests, we can add:

require 'sidekiq/testing'

module Sidekiq::Worker::ClassMethods
  def execute_job(worker, args)
    Sidekiq::Status::ServerMiddleware.new.call(worker, nil, nil) do
      worker.perform(*args)
    end
  end
end

So, I would like to know if this is a valid way to test this gem? And, if so, would you be willing to accept a PR? Also, I am still not sure how to test nil status with this, when the job expires after the given expiration time.

module Sidekiq
module Status
class << self
def status(jid)
:complete
end
def message(jid)
""
end
end
module Storage
def store_for_id(id, status_updates, expiration = nil)
end
def read_field_for_id(id, field)
end
end
end
end

Allows at command to be used during testing .. goes no where but with inline blocks and no redis code in test cases can still be executed.

Hello

First thanks a lot for the gem. I would like to know if it will be possible to avoid crashing when redis is down. If redis is down and sidekiq status is installed it crash at the first call (because of the middleware settings). I gave a quick look without founding the proper answer to patch it.

Is making check after method call of redis connection will be a good idea? It's seems quite dirty to my point.

The problem is yesterday our redis was buggy and the server crashed a lot because of Sidekiq-Status trying to connect to redis and crashing. 😞

Thanks

our sidekiq.rb looks like this:

require 'sidekiq'
require 'sidekiq-status'
require 'resolv-replace' unless Rails.env.test?

redis = proc { Redis.new(url: RedisUtil.redis_url('', 'sidekiq_db')) }

Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 25, &redis)
  config.server_middleware do |chain|
    chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes # default
  end
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes # default
  end
end

Sidekiq.configure_client do |config|
  config.redis = ConnectionPool.new(size: 25, &redis)
  config.client_middleware do |chain|
    chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes # default
  end
end
Sidekiq.default_worker_options = { 'backtrace' => 5, 'retry' => 3 }

Also our routes for sidekiq looks like:

require 'sidekiq/web'
require 'sidekiq-status/web'
authenticate :admin do
  mount Sidekiq::Web => '/sidekiq'
end

We also added include Sidekiq::Status::Worker in job.rb

Great gem :)

I was just wondering about the best way to get/set the status of a job outside the scope of a worker. I notice that the Sidekiq::Status.store_for_id() is protected, so can't be called statically, say from the scope of an MVC model. So I've just made my own helper like this:

module Sidekiq::Status
  class << self
    def broadcast jid, status_updates
        Sidekiq.redis do |conn|
          conn.multi do
            conn.hmset  jid, 'update_time', Time.now.to_i, *(status_updates.to_a.flatten(1))
            conn.expire jid, Sidekiq::Status::DEFAULT_EXPIRY
            conn.publish "status_updates", jid
          end[0]
        end
    end
  end
end

It can't inherit any of the overrides to expiration but apart from that it seems to be working well so far. Just wanting some feedback on this really...

When I look at redis with redis-browser, I see that job status are stored directly at root with the jid as key... I think it makes more sense to put them in some sidekiq:status: container.

The current way makes browsing redis a pain.

Hi there,

I'd like to open a PR for this, but I wanted to double check with you first if you think it makes sense.

We use zrangebyscore to look for the job in the schedule sorted set. That operation has a complexity of:

O(log(N)+M) with N being the number of elements in the sorted set and M the number of elements being returned. If M is constant (e.g. always asking for the first 10 elements with LIMIT), you can consider it O(log(N)).

After that weuse zrem with a complexity of:

O(M*log(N)) with N being the number of elements in the sorted set and M the number of elements to be removed.

Since we're removing 1 element, it becomes O(log(N)).

We look for the job in the schedule set to know if we can remove it, but zrem returns the number of removed elements, so we could use that to decide if the element was there and just delete it. Is that right?

I have a use case where we have quite a few scheduled jobs that I need to unschedule sometimes, and if I didn't need to keep track of the timestamp it would simplify things a bit.

The functionality of this gem really looks nice and handy, however I'm lacking a bit of information on the production-readyness of the gem. Sidekiq is very high performing, and is often used as a better alternative to resque due to resource issues. Now, how much additional overhead comes from using this status gem? Is it mainly redis lookups or ruby processing time? Is anyone using it for processing hundred of 1000's jobs everyday? How many extra calls to redis is it adding on top of the ones performed by sidekiq?

Hope it's okay that I ask, because I will have a hard time simulating my production setup to test the performance locally.

Will any of the status methods work for jobs scheduled to run after the expiration time? In the readme it says "Query for job status any time later:" which makes it seem like the status will always be available. After doing some investigation, it appears that this gem works by storing a hash in redis with info about the job status and the hash ttl is the config expiration time. If this is true, I think the docs should be revised to have a clearer explanation of what the expiration time is used for as well as mentioning the impact of the expiration time on scheduled jobs in the Retrieving Status section.
Thanks

Since version 2.2.0 Sidekiq creates and returns a JID when you call .perform_async. So you potentially wouldn't have to do that in your gem.

https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/client.rb#L21

Really like this gem, do you plan on updating it?

Hi,
we would like to use sidekiq-status in our environment, at the moment when we pull we don't get the newest code without the limitation of sidekiq gem.
Could you please tag and publish a new version?

Thanks

Currently this gem adds top-level keys, for example
"hmset" "a3621311f0aef8626935b249" "update_time" "1428675833" "status" "queued"

they pollute the redis keyspace. I suggest to namespace them, like
'sidekiq-status:a3621311f0aef8626935b249'

Would you be against storing the backtrace for failed jobs so they can be retrieved within the timeout period?

Something like Sidekiq::Status.backtrace(job_id)?

I'd like to query SideKiq::Status to get all the 'working' jobs. Is this possible?

Thanks

Sidekiq 4 was released a a few weeks ago, but we can't upgrade as sidekiq-status is locked to >= 2.7

The following commit changes the client_push method in testing environment so as it returns an job id.

sidekiq/sidekiq@8dd3660

Would be nice if sidekiq-status could retrieve jobs from the testing environment as well, instead of looking into Redis.

Sidekiq::Status.unschedule(nil) unschedules the first scheduled job (no matter what it is). Unless I'm missing something here, I think the wanted behaviour is to just return nil without acting (or even returning an error).