hugopl / sidekiq.cr

Simple, efficient job processing for Crystal

Sidekiq.cr

Sidekiq is a well-regarded background job framework for Ruby. Now we're bringing the awesomeness to Crystal, a Ruby-like language. Why? To give you options. Ruby is friendly and flexible but not terribly fast. Crystal is statically-typed, compiled and very fast but retains a similar syntax to Ruby.

Rough, initial benchmarks on macOS 10.14.5, ruby 2.7.2:

Runtime	RSS	Time	Throughput
Sidekiq 6.2.0	55MB	16.4	6,100 jobs/sec
Sidekiq 6.2.0/hiredis	49MB	13.0	7,900 jobs/sec
Crystal 0.35.1	15MB	3.8	26,000 jobs/sec

If you have jobs which are CPU-intensive or require very high throughput, Crystal is an excellent alternative to native Ruby extensions. It compiles to a single executable so deployment is much easier than Ruby.

Getting Started

Please see the wiki for in-depth documentation and how to get started using Sidekiq.cr in your own app.

Support

Sidekiq.cr is community-supported and not commercially supported by @mperham and Contributed Systems. General maintenance and bug fixes are always welcomed.

Help wanted

See the issues for chores and other ideas to help.

Things that do not exist and probably won't ever:

• Support for daemonization, pidfiles, log rotation - use Upstart/Systemd
• Delayed extensions - too dynamic for Crystal

The Ruby and Crystal versions of Sidekiq must remain data compatible in Redis. Both versions should be able to create and process jobs from each other. Their APIs are not and should not be identical but rather idiomatic to their respective languages.

Naming

Sidekiq is a registered trademark of Contributed Systems who has granted use of the name to this project.

Thanks

Originally developed by Mike Perham, http://www.mikeperham.com. Maintained by Hugo Parente Lima.

sidekiq.cr's Issues

Error resolving dependencies

Hi,

I can see that in shards.yml there is :

kemal:
  github: sdogruyol/kemal
  version: ~> 0.14.0

I use kemal as a dependencies but in 0.18.0.

PR : #39

crashes with Crystal 0.18

hey,

first of all, thanks for all the work and porting sidekiq to crystal. i have a little project where i port a tiny Sinatra app to Sinatra. (no worries, it's not in production yet) i upgraded to Crystal 0.18 and noticed that 0.18 doesn't like Sidekiq.

not sure if you are aware of this :)

crystal run src/lighthouse.cr              
Bug: trying to upcast Bool <- Sidekiq::Context+
[4557775923] *raise<String>:NoReturn +163
[4566977509] *Crystal::CodeGenVisitor#upcast_distinct<Crystal::CodeGenVisitor, LLVM::Value, Crystal::Type+, Crystal::Type+>:NoReturn +437
[4566966598] *Crystal::CodeGenVisitor#upcast<Crystal::CodeGenVisitor, LLVM::Value, Crystal::Type+, Crystal::Type+>:LLVM::Value +1334
[4567065912] *Crystal::CodeGenVisitor::Phi#add<Crystal::CodeGenVisitor::Phi, LLVM::Value, Crystal::Type+, Bool>:LLVM::Value? +264
[4567087844] *Crystal::CodeGenVisitor#codegen_if_branch<Crystal::CodeGenVisitor, Crystal::CodeGenVisitor::Phi, Crystal::ASTNode+, LLVM::BasicBlock, Bool>:LLVM::Value? +116
[4566997261] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +18381
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4567062874] *Crystal::CodeGenVisitor#target_def_fun<Crystal::CodeGenVisitor, Crystal::Def+, Crystal::Type+>:LLVM::Function +2426
[4567110500] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +580
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4566983396] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +4516
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4567062874] *Crystal::CodeGenVisitor#target_def_fun<Crystal::CodeGenVisitor, Crystal::Def+, Crystal::Type+>:LLVM::Function +2426
[4567110500] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +580
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4567066435] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Assign>:Bool? +259
[4566985501] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +6621
[4566983396] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +4516
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4567062874] *Crystal::CodeGenVisitor#target_def_fun<Crystal::CodeGenVisitor, Crystal::Def+, Crystal::Type+>:LLVM::Function +2426
[4567116337] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +6417
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4566983396] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +4516
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4567062874] *Crystal::CodeGenVisitor#target_def_fun<Crystal::CodeGenVisitor, Crystal::Def+, Crystal::Type+>:LLVM::Function +2426
[4567116337] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +6417
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4567062874] *Crystal::CodeGenVisitor#target_def_fun<Crystal::CodeGenVisitor, Crystal::Def+, Crystal::Type+>:LLVM::Function +2426
[4567116337] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +6417
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4566983396] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +4516
[4567055030] *Crystal::CodeGenVisitor#codegen_fun<Crystal::CodeGenVisitor, String, Crystal::Def+, Crystal::Type+, Bool, LLVM::Module, Bool, Bool>:LLVM::Function +3414
[4566993639] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +14759
[4567110515] *Crystal::CodeGenVisitor#visit<Crystal::CodeGenVisitor, Crystal::Call>:Bool +595
[4566980651] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +1771
[4566983396] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +4516
[4566985894] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +7014
[4566979292] *Crystal::ASTNode+@Crystal::ASTNode#accept<Crystal::ASTNode+, Crystal::CodeGenVisitor>:Nil +412
[4566833098] *Crystal::Compiler#codegen<Crystal::Compiler, Crystal::Program, Crystal::Expressions, Array(Crystal::Compiler::Source), String>:Array(String)? +5690
[4558732176] *Crystal::Compiler#compile<Crystal::Compiler, Array(Crystal::Compiler::Source), String>:Crystal::Compiler::Result +4608
[4567585015] *Crystal::Command#run_command<Crystal::Command, Bool>:Nil +151
[4557948526] *Crystal::Command#run<Crystal::Command>:(Array(Crystal::ImplementationTrace) | Array(Crystal::Init::View+:Class) | Array(String) | Bool | Crystal::Compiler::Result | Hash(String, String) | IO::FileDescriptor+ | Nil) +6398
[4557861057] main +17505

Error: you've found a bug in the Crystal compiler. Please open an issue, including source code that will allow us to reproduce the bug: https://github.com/crystal-lang/crystal/issues

i will try to do some debugging, wanted to document this here just in case somebody else runs in the same problem.

thanks
ben

Sidekiq.cr breaks "Busy" tab for Ruby Web UI

First of all, thanks for Sidekiq.cr - wouldn't have tried Crystal without it!

There seems to be an issue with the "Busy"-tab for the Ruby Web UI when running Sidekiq.cr:

TypeError at /sidekiq/busy can't convert String into an exact number file: calculations.rb location: at line: 45

This seems to be related to https://github.com/mperham/sidekiq/blob/master/web/views/busy.erb#L90.

If I replace <%= relative_time(Time.at(msg['run_at'])) %> with <%= msg['run_at'] %> the "Busy" tab works again.

If I take one of the timestamps printed in the "Started" column and run it in the ruby console with Time.at:
$ Time.at("1466511737")
TypeError: can't convert String into an exact number

So it seems that the Sidekiq Web UI for Ruby can't handle the data for 'run_at' inserted from Sidekiq.cr?

Web Dashboard

Does this Sidekiq version have the ruby/rails dashboard? If so, any documentation on the routes? If not, any plans to add this?

Add Data API

Sidekiq's Data API allows users to mine Redis for their job data. Create a Crystal version of it.

Fix broken ascii art

Blocked, waiting on crystal-lang/crystal@5e04f9f to land in a release.

Update redis to new version

While trying to run shards install, after adding sidekiq dependency, the console shows this message:
Error resolving redis (~> 1.8.0, ~> 2.0)

Redis dependency now shows another version, and the issue disapears by removing sidekiq's lines.

dependencies:
  redis:
    github: stefanwille/crystal-redis
    version: ~> 2.5.3

Current:

dependencies:
  redis:
    github: stefanwille/crystal-redis
    version: ~> 2.0

Thanks!

Findings while working through 'Getting Started'

Hi mperham,

I confess I am a total newbie to crystal. Yesterday was the first time I dabbled with it in any way.

I've worked my way through the 'Getting Started' page for sidekiq very literally. Meaning, that I do exactly what the text specifies. The points below are the issues I found along the way.

I kindly request that the text of the page gets adjusted, so that newbies like me have a chance getting the examples to work.

Redis needs to be installed

This may sound obvious, but is not mentioned anywhere. Also, a reference to getting and installing it may be appropriate.

Example sidekiq.cr doesn't output anything

I got it running as specified by text, but no output from the workers ensued.
Only after downloading the examples and running the sidekiq.cr did I get anything useful. I surmise that the code difference between the example code and the code as given in the text is significant.
Please update the code in the text so that it works in one go.

Example web.cr doesn't compile

This gives the following error:

Error opening file '/home/wim/projects/try/crystal/try/sample/libs/sidekiq/sidekiq/../../web/views/dashboard.ecr' with mode 'r': No such file or directory (Errno)

Only after an irc consult did it become apparent to me that certain assets where missing for web.cr.

Repo examples work fine locally

In the end, I gave up, cloned the repo and ran the examples from there. This worked perfectly, but nullifies the content of the 'Getting Started' page. It might as well not exist.

Besides the above points, I would suggest a 'Quick Start' at the top of the 'Getting Started' page, with terse instructions as to how to run the repo examples.

---

Thank you for your attention; I hope it's clear I'm not trying to critique here. I simply wish for my successors to have a better experience.

Port to Crystal 1.0.0

The issue is just on resolving shard dependencies, so better wait some days while some dependencies get releases for Crystal 1.0.0, so we can do the same... update the dependencies then release.

Issue: https://travis-ci.org/github/mperham/sidekiq.cr/builds/764020171

Logger::INFO undefined(?) with assignment of Sidekiq::Client.default_context

Issue with assignment of default context in personal Crystal app, however when adding an example spec to sidekiq.cr all passing.

Following the instructions from this Wiki entry.

https://github.com/mperham/sidekiq.cr/wiki/Getting-Started#create-some-jobs-using-the-client-api

$ crystal src/tasks/async_price.cr playground/price.csv
Error in ./src/tasks/async_price.cr:22: instantiating 'Sidekiq::Client::Context:Class#new()'

Sidekiq::Client.default_context = Sidekiq::Client::Context.new
                                                           ^~~

instantiating 'Sidekiq::Client::Context#initialize()'

in ./libs/sidekiq/sidekiq/client.cr:16: instantiating 'Sidekiq::Logger:Class#build()'

        @logger = Sidekiq::Logger.build
                                  ^~~~~

instantiating 'build(IO::FileDescriptor)'

in ./libs/sidekiq/sidekiq/logger.cr:48: undefined constant Logger::INFO

      logger.level = Logger::INFO

$ crystal --version
Crystal 0.19.0 (2016-09-02)

Example of spec/client_spec.cr passing.

describe "Sidekiq::Client" do
  it "must allow assignment for a default context" do
    Sidekiq::Client.default_context = Sidekiq::Client::Context.new
    Sidekiq::Client.default_context.class.should eq(Sidekiq::Client::Context)
  end
end

Remove DUMP binary fixtures

These don't work across the 3.0 / 3.2 boundary. Create the fixture data programmatically or remove the tests completely.

sidekiq_options

The sidekiq_options method for setting arbitrary worker options doesn't exist. Rethink how this could be done.

class SomeWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'foo', unique_for: 10.minutes

Namespace support

Sidekiq in Ruby supports passing a namespace key to the redis configuration. It would be great if the same was possible in this version of Sidekiq. I've read your blog post about your preference for redis DBs instead of namespaces, but I think redis DBs are very unreadable and it's easy to make a costly mistake.

Redis early access

I've found a bug with Sidekiq Worker and Redis :

If Redis is listening on something else than localhost Sidekiq Worker will fail to start with an error : Socket::ConnectError: Error connecting to 'localhost:6379': Connection refused

I tried to configure it manually with :

# Create CLI
cli = Sidekiq::CLI.new

# Create worker
worker = cli.create
worker.redis = Sidekiq::RedisConfig.new(**redis_config)

but it breaks at worker = cli.create and worker.redis = Sidekiq::RedisConfig.new(**redis_config) is never reached.

It thinks it's due to https://github.com/mperham/sidekiq.cr/blob/master/src/sidekiq/server/server.cr#L40 which creates a Pool which creates a RedisConfig with default values that cannot work.

I know it can be worarounded by using env vars but what if I don't want to use it? (I already have a configuration file for my application)

Broken pipe after Redis connection outage

Hi,

First of all thank you for the fantastic sidekiq library! The crystal version proves to be very valuable in our Docker/Kubernetes environment with it's low resource consumption and simple single binary deployment.

I see an issue under the condition that the Redis connection is temporary unavailable (i.e. restart of standalone Redis instance) - Sidekiq hangs in the "heartbeat" loop and is unable to recover after Redis is back online:

2018-03-27T06:57:08.645Z 10290 TID-204vbeo  INFO: Sidekiq v0.6.1 in Crystal 0.24.1
2018-03-27T06:57:08.646Z 10290 TID-204vbeo  INFO: Licensed for use under the terms of the GNU LGPL-3.0 license.
2018-03-27T06:57:08.646Z 10290 TID-204vbeo  INFO: Upgrade to Sidekiq Enterprise for more features and support: http://sidekiq.org
2018-03-27T06:57:08.646Z 10290 TID-204vbeo  INFO: Starting processing with 1 workers
2018-03-27T06:57:08.646Z 10290 TID-204vbeo  INFO: Press Ctrl-C to stop
2018-03-27T06:57:13.159Z 10290 TID-204vaww  ERROR: Error fetching job: RedisError: Disconnected
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis/connection.cr:0:7 in 'receive_line'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis/connection.cr:74:5 in 'receive'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis/strategy/single_statement.cr:12:5 in 'command'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis.cr:219:5 in 'command'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis/command_execution/value_oriented.cr:67:9 in 'array_or_nil_command'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/redis/src/redis/commands.cr:864:7 in 'brpop'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/sidekiq/src/sidekiq/server/fetch.cr:54:37 in 'retrieve_work'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/sidekiq/src/sidekiq/server/processor.cr:64:16 in 'get_one'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/sidekiq/src/sidekiq/server/processor.cr:73:7 in 'fetch'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/sidekiq/src/sidekiq/server/processor.cr:57:14 in 'process_one'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: lib/sidekiq/src/sidekiq/server/processor.cr:48:11 in 'run'
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: ~procProc(Nil)@lib/sidekiq/src/sidekiq/server/processor.cr:39
2018-03-27T06:57:13.330Z 10290 TID-204vaww  ERROR: ~procProc(Nil)@lib/sidekiq/src/sidekiq/server/util.cr:20
2018-03-27T06:57:13.331Z 10290 TID-204vaww  ERROR: /usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:255:3 in 'run'
2018-03-27T06:57:13.331Z 10290 TID-204vaww  ERROR: ~proc2Proc(Fiber, (IO::FileDescriptor | Nil))@/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:29
2018-03-27T06:57:13.817Z 10290 TID-204vaps  ERROR: heartbeat: RedisError: Disconnected
2018-03-27T06:57:18.818Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
2018-03-27T06:57:21.524Z 10290 TID-204vatc  WARN: Errno: Error writing to socket: Broken pipe
2018-03-27T06:57:21.530Z 10290 TID-204vatc  WARN: /usr/local/Cellar/crystal-lang/0.24.1_2/src/socket.cr:72:11 in 'unbuffered_write'
/usr/local/Cellar/crystal-lang/0.24.1_2/src/io/buffered.cr:184:5 in 'flush'
lib/redis/src/redis/connection.cr:38:5 in 'flush'
lib/redis/src/redis/connection.cr:30:5 in 'send'
lib/redis/src/redis/strategy/single_statement.cr:11:5 in 'command'
lib/redis/src/redis.cr:219:5 in 'command'
lib/redis/src/redis/command_execution/value_oriented.cr:43:9 in 'string_array_command'
lib/redis/src/redis/commands.cr:1232:7 in 'zrangebyscore'
lib/redis/src/redis/commands.cr:1224:5 in 'zrangebyscore:limit'
lib/sidekiq/src/sidekiq/server/scheduled.cr:20:25 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:8:7 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:79:11 in 'enqueue'
~procProc(Nil)@lib/sidekiq/src/sidekiq/server/scheduled.cr:65
~proc2Proc(Nil)@lib/sidekiq/src/sidekiq/server/util.cr:20
/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:255:3 in 'run'
~proc2Proc(Fiber, (IO::FileDescriptor | Nil))@/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:29
2018-03-27T06:57:23.821Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
2018-03-27T06:57:28.823Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
2018-03-27T06:57:31.332Z 10290 TID-204vatc  WARN: Errno: Error writing to socket: Broken pipe
2018-03-27T06:57:31.338Z 10290 TID-204vatc  WARN: /usr/local/Cellar/crystal-lang/0.24.1_2/src/socket.cr:72:11 in 'unbuffered_write'
/usr/local/Cellar/crystal-lang/0.24.1_2/src/io/buffered.cr:184:5 in 'flush'
lib/redis/src/redis/connection.cr:38:5 in 'flush'
lib/redis/src/redis/connection.cr:30:5 in 'send'
lib/redis/src/redis/strategy/single_statement.cr:11:5 in 'command'
lib/redis/src/redis.cr:219:5 in 'command'
lib/redis/src/redis/command_execution/value_oriented.cr:43:9 in 'string_array_command'
lib/redis/src/redis/commands.cr:1232:7 in 'zrangebyscore'
lib/redis/src/redis/commands.cr:1224:5 in 'zrangebyscore:limit'
lib/sidekiq/src/sidekiq/server/scheduled.cr:20:25 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:8:7 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:79:11 in 'enqueue'
~procProc(Nil)@lib/sidekiq/src/sidekiq/server/scheduled.cr:65
~proc2Proc(Nil)@lib/sidekiq/src/sidekiq/server/util.cr:20
/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:255:3 in 'run'
~proc2Proc(Fiber, (IO::FileDescriptor | Nil))@/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:29
2018-03-27T06:57:33.828Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
2018-03-27T06:57:38.831Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
2018-03-27T06:57:38.884Z 10290 TID-204vatc  WARN: Errno: Error writing to socket: Broken pipe
2018-03-27T06:57:38.891Z 10290 TID-204vatc  WARN: /usr/local/Cellar/crystal-lang/0.24.1_2/src/socket.cr:72:11 in 'unbuffered_write'
/usr/local/Cellar/crystal-lang/0.24.1_2/src/io/buffered.cr:184:5 in 'flush'
lib/redis/src/redis/connection.cr:38:5 in 'flush'
lib/redis/src/redis/connection.cr:30:5 in 'send'
lib/redis/src/redis/strategy/single_statement.cr:11:5 in 'command'
lib/redis/src/redis.cr:219:5 in 'command'
lib/redis/src/redis/command_execution/value_oriented.cr:43:9 in 'string_array_command'
lib/redis/src/redis/commands.cr:1232:7 in 'zrangebyscore'
lib/redis/src/redis/commands.cr:1224:5 in 'zrangebyscore:limit'
lib/sidekiq/src/sidekiq/server/scheduled.cr:20:25 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:8:7 in 'enqueue_jobs'
lib/sidekiq/src/sidekiq/server/scheduled.cr:79:11 in 'enqueue'
~procProc(Nil)@lib/sidekiq/src/sidekiq/server/scheduled.cr:65
~proc2Proc(Nil)@lib/sidekiq/src/sidekiq/server/util.cr:20
/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:255:3 in 'run'
~proc2Proc(Fiber, (IO::FileDescriptor | Nil))@/usr/local/Cellar/crystal-lang/0.24.1_2/src/fiber.cr:29
2018-03-27T06:57:43.834Z 10290 TID-204vaps  ERROR: heartbeat: Error writing to socket: Broken pipe
...

I would expect sidekiq.cr either to reconnect and continue or to throw an error and exit.

Thanks,
Nik

Easy to swap with Ruby version of sidekiq?

I am aware of the README note:

The Ruby and Crystal versions of Sidekiq must remain data compatible in Redis. Both versions should be able to create and process jobs from each other. Their APIs are not and should not be identical but rather idiomatic to their respective languages.

I'm not experienced with Ruby but I am interested in a Ruby project(Discourse) where Sidekiq is used. I haven't gone over the codebase to look over how their using the API(or aware of what the API differences to look out for are).

From what I understand Sidekiq is run separate to the main Ruby project, so I was curious if that meant Sidekiq.cr might be potentially able to be a drop-in replacement in some cases?

Retry job using job id

Is it possible to retry a job if I only have a job id?

Handle complex arguments

The perform_types macro handles the marshaling of JSON args to the perform method but can't handle Arrays or Hashes yet, it needs to properly recurse.

Add Web UI

Sidekiq's Web UI is based on Sinatra. Investigate building a native Crystal version using Kemal and ecr views instead.

Requires #11.

Looking for examples of sidekiq.cr used in production with a Rails app

I’m very excited by the prospect of running Sidekiq.cr besides Sidekiq.rb to handle compute-heavy jobs.

But I’m struggling to find any real world user accounts of this being done. Has anyone successfully sped up their application with this and, better yet, blogged about it?

Package web assets for easy deployment

Right now the files in web/assets are served from the filesystem, which means they need to be deployed too. That's a PITA. Investigate using a virtual filesystem of resources embedded within the binary.

Fix float serialization

Sidekiq.rb stores timestamps as epoch floats in JSON. Crystal's Float#to_s is broken in this regard so we need to do some custom serialization to maintain compatibility.

crystal-lang/crystal#2643

Move away from JSON.mapping

We need to be able to round-trip arbitrary hash entries. JSON.mapping will lose unknown entries when it skips them. Roll our own conversion from Sidekiq::Job <-> Hash.

Add Fiber-local storage

Things like the logger need to print job-specific context with each line. To do that, sidekiq.rb uses thread-local storage. Crystal doesn't supply fiber-local storage so grow our own.

Release latest changes

Could you provide a release of the latest changes?

Missing CSRF protection

Kemal doesn't support sessions yet so we can't implement CSRF protection via a session-based authenticity token. We need to implement this once session support has been released.

Sidekiq web UI compiling error due to name space changed in kemal-session

When I tried to compile the example provided in Sidekiq.cr, I encountered the following error:

src/web.cr:5: undefined constant Session

Session.config do |config|
^~~~~~~

Trace the shards.yml in Sidekiq.cr, I found cause.

Sidekiq.cr depended kemal-crsf 0.2.0 which depended on kemal-session master branch. However, one of the recent changes in kemal-session master branch was to change the name space for kemal session from "Session" to "Kemal::Session". That caused the above undefined constant error on "Session". I added a class alias before the Session.config section:

Session = Kemal::Session

and that walked around the issue for the time being. However, I think it would be better to fix this from the root.

Then, I checked kemal-crsf and found it had a new tag 0.3.0 created which specified kemal-session dependency to version 0.8.0 explicitly instead of the master branch. So, I guess in Sidekiq.cr, the kemal-crsf dependency should be upgraded to this tag. On the other hand, I also noticed kemal-csrf 0.3.0 upgraded its kemal to 0.20.0 while Sidekiq still used 0.19.0. This could be a potential conflict. You might want to upgrade Sidekiq to the most recent kemal version to 0.2.0.

I am still new to Crystal so my suggestions could be incorrect. just want to share my findings and hope this could help fix the issue quickly. Thank you very much for porting Sidekiq to Crystal which is a superb job scheduling product!

restrict type on sender too

class MyWorker
  include Sidekiq::Worker

  def perform(x : Int64)
    logger.info "hello!"
  end
end

MyWorker.async.perform(1_i64)
MyWorker.async.perform("bla")

compiled, but expected compile error

Periodic jobs

Is there any way to have periodic jobs?

Build

We need a build!

Travis must build fixtures

Before running the test suite, Travis needs to install the sidekiq gem and run make fixtures.

Run worker with perform_at

Hello Mike,
I'm trying to launch worker
OkPhotos::TopPhotoUpdateWorker.async.perform_at(2.minutes.from_now, group)

but I've got an error:

instantiating '_perform_at(Time, String)'
in ./libs/sidekiq/sidekiq/job.cr:87: no overload matches 'OkPhotos::TopPhotoUpdateWorker::PerformProxy#perform_in' with types Float64, String
Overloads are:
 - OkPhotos::TopPhotoUpdateWorker::PerformProxy#perform_in(interval : Time::Span, group : Hash(String, Array(String) | String))

      perform_in(interval.epoch_f, args)
      ^~~~~~~~~~

perform_in works well.

If I do something wrong? Sorry, it's my first issue)

format code

with crystal tool format src/ spec/ bench/

Implement setproctitle

Sidekiq.rb has a nice proctitle integration. Resurrect it based on mailing list suggestion.

https://groups.google.com/forum/?fromgroups#!topic/crystal-lang/cF-wWINyj_E

Sidekiq web example is not compiling

Clean up boot API

Sidekiq needs to expose an API for the Client, Web UI and Server binaries, e.g. here. Right now the API is less than perfect. Spend some time designing something cleaner.

No need for `perform_types`

I'm having some troubles having the specs pass (do I need a specific redis version?) but one thing you can do to simplify the definition of workers is to put this in the macro included of Sidekiq::Worker:

macro method_added(a_def)
  \{% if a_def.name.id == "perform".id %}
    def _perform(args : Array(JSON::Type))
      perform(
        \{% for arg, index in a_def.args %}
          \{% if arg.restriction %}
            args[\{{index}}].as(\{{arg.restriction}}),
          \{% else %}
            \{{ raise "argument '#{arg}' must have a type restriction" }}
          \{% end %}
        \{% end %}
      )
    end
  \{% end %}
end

And the remove the perform_types macro. This will make it so that when a user defines perform, the macro will automatically define _perform using perform's arguments and types. As a bonus, if you don't put a type restriction in an argument you can a compile error.

negative count (ArgumentError) with Crystal 0.20.3

When running examples/sidekiq.cr with Crystal 0.20.3 i get the following exception:

2017-01-11T11:41:06.509Z 3239 TID-6tq4g  INFO: Pushing job 60d463fda326627ddfb54268
negative count (ArgumentError)
0x4864c7: *CallStack::unwind:Array(Pointer(Void)) at ??
0x48645a: *CallStack#initialize:Array(Pointer(Void)) at ??
0x48642a: *CallStack::new:CallStack at ??
0x46f8be: *raise<ArgumentError>:NoReturn at ??
0x499ef2: *String#byte_slice<Int32, Int32>:String at ??
0x50cbf5: *Redis::Connection#receive_line:String at ??
0x50c691: *Redis::Connection#receive:(Array(Redis::RedisValue) | Int64 | String | Nil) at ??
0x5149f3: *Redis::Strategy::Transaction#single_command<Array(String)>:(Array(Redis::RedisValue) | Int64 | String | Nil) at ??
0x514b4b: *Redis::Strategy::Transaction#exec:Array(Redis::RedisValue) at ??
0x514a58: *Redis::Strategy::Transaction#commit:Array(Redis::RedisValue) at ??
0x51c4e4: *Sidekiq::Client#raw_push<Array(CrystalWorker::PerformProxy)>:Bool at ??
0x51c2b7: *Sidekiq::Client#push<CrystalWorker::PerformProxy>:(String | Nil) at ??
0x51d55a: *CrystalWorker::PerformProxy at ??
0x51d535: *CrystalWorker::PerformProxy#perform<Int64>:(String | Nil) at ??
0x46f3db: ??? at ??
0x47db69: main at ??
0x7f1941d7c830: __libc_start_main at ??
0x46e789: _start at ??
0x0: ??? at ??

This issue is fixed in crystal-redis 1.7.1: stefanwille/crystal-redis@8d7e448

Can't run example in manjaro linux

Unhandled exception: Missing ENV key: "" (KeyError)
from /usr/lib/crystal/env.cr:53:7 in 'fetch'
from /usr/lib/crystal/env.cr:22:5 in '[]'
from lib/sidekiq/src/sidekiq/pool.cr:27:9 in 'initialize'
from lib/sidekiq/src/sidekiq/pool.cr:25:5 in 'initialize:pool_size'
from lib/sidekiq/src/sidekiq/pool.cr:25:5 in 'new:pool_size'
from lib/sidekiq/src/sidekiq/pool.cr:61:7 in 'initialize'
from lib/sidekiq/src/sidekiq/pool.cr:60:5 in 'new'
from lib/sidekiq/src/sidekiq/server/server.cr:40:15 in 'initialize'
from lib/sidekiq/src/sidekiq/server/server.cr:24:5 in 'initialize:concurrency:queues:environment:logger'
from lib/sidekiq/src/sidekiq/server/server.cr:24:5 in 'new:concurrency:queues:environment:logger'
from lib/sidekiq/src/sidekiq/server/cli.cr:45:7 in 'create'
from lib/sidekiq/src/sidekiq/server/cli.cr:52:7 in '__crystal_main'
from /usr/lib/crystal/crystal/main.cr:97:5 in 'main_user_code'
from /usr/lib/crystal/crystal/main.cr:86:7 in 'main'
from /usr/lib/crystal/crystal/main.cr:106:3 in 'main'
from __libc_start_main
from _start
from ???

require only works with -> require "../src/sample/workers.cr"

Optimize logging output

I noticed that setting the benchmark logger level to WARN, the jobs per sec went from 15k to 36k!

To reproduce:

1. Run make bench.
2. Edit bench/load.cr and change the log level on line 26 from INFO to WARN.
3. Repeat 1.

I optimized the Logger::Formatter here to stream to the IO but it didn't appear to make a difference. The logger is used here. Any ideas on how to minimize the log overhead?

Update to support 0.25.0

With many changes to JSON and File classes, this shard is broken in 0.25.0

Update Wiki - Basic auth example

https://github.com/mperham/sidekiq.cr/wiki/Getting-Started#the-web-ui (to my knowledge you cannot create pr to a wiki, right? 😅)

Kemal.config do |config|
  # To enable SSL termination:
  # ./web --ssl --ssl-key-file your_key_file --ssl-cert-file your_cert_file
  #
  # For more options, including changing the listening port:
  # ./web --help
  #
  # Basic authentication:
  #
  # config.add_handler Kemal::Middleware::HTTPBasicAuth.new("username", "password")
end

should be probably something like:

# Basic authentication:
#
# require "kemal-basic-auth"
# basic_auth "username", "password"

Kemal.config do |config|
  # To enable SSL termination:
  # ./web --ssl --ssl-key-file your_key_file --ssl-cert-file your_cert_file
  #
  # For more options, including changing the listening port:
  # ./web --help
end

References:
https://github.com/kemalcr/kemal-basic-auth#usage
kemalcr/kemal@b4c10a3

Invalid type derivation

Somehow the Crystal compiler thinks that Sidekiq::Job#retry can be of type Array(Redis::RedisValue) but I clearly mark its type here. I can't figure out what is wrong. Can anyone help?

    property retry : (Bool | Int64 | Nil)

    def to_h : Hash(String, JSON::Type)
      h = Hash(String, JSON::Type).new
      h["jid"] = jid
      h["args"] = args
      h["queue"] = queue
      h["class"] = klass
      h["created_at"] = created_at.epoch_f

      h["bid"] = bid if bid
      h["at"] = at.not_nil!.epoch_f if at
      h["failed_at"] = failed_at.not_nil!.epoch_f if failed_at
      h["retried_at"] = retried_at.not_nil!.epoch_f if retried_at
      h["enqueued_at"] = enqueued_at.not_nil!.epoch_f if enqueued_at
      h["retry_count"] = retry_count if retry_count
      h["dead"] = dead if dead
      h["backtrace"] = backtrace if backtrace
      h["retry"] = retry if retry
      h["error_class"] = error_class if error_class
      h["error_message"] = error_message if error_message
      h["error_backtrace"] = error_backtrace.not_nil!.map{|x| x.as(JSON::Type)} if error_backtrace
      h
    end

$ crystal spec
Error in ./spec/api_spec.cr:136: instantiating 'Sidekiq::SortedEntry#add_to_queue()'

      job.add_to_queue
          ^~~~~~~~~~~~

in ./src/sidekiq/api.cr:370: instantiating 'remove_job()'

      remove_job do |message|
      ^~~~~~~~~~

in ./src/sidekiq/api.cr:407: instantiating 'Sidekiq:Module#redis()'

      Sidekiq.redis do |conn|
              ^~~~~

in ./src/sidekiq.cr:15: instantiating 'Sidekiq::Pool#redis()'

    Sidekiq::Client.default_context.pool.redis do |conn|
                                         ^~~~~

in ./src/sidekiq/pool.cr:65: instantiating 'ConnectionPool(Redis)#connection()'

      @pool.connection do |conn|
            ^~~~~~~~~~

in ./src/sidekiq/pool.cr:65: instantiating 'ConnectionPool(Redis)#connection()'

      @pool.connection do |conn|
            ^~~~~~~~~~

in ./src/sidekiq.cr:15: instantiating 'Sidekiq::Pool#redis()'

    Sidekiq::Client.default_context.pool.redis do |conn|
                                         ^~~~~

in ./src/sidekiq/api.cr:407: instantiating 'Sidekiq:Module#redis()'

      Sidekiq.redis do |conn|
              ^~~~~

in ./src/sidekiq/api.cr:370: instantiating 'remove_job()'

      remove_job do |message|
      ^~~~~~~~~~

in ./src/sidekiq/api.cr:374: instantiating 'Sidekiq::Client#push(Sidekiq::Job)'

        Sidekiq::Client.new.push(job)
                            ^~~~

in ./src/sidekiq/client.cr:100: instantiating 'raw_push(Array(Sidekiq::Job+))'

        raw_push([job])
        ^~~~~~~~

in ./src/sidekiq/client.cr:135: instantiating 'Sidekiq::Pool#redis()'

      @ctx.pool.redis do |conn|
                ^~~~~

in ./src/sidekiq/pool.cr:65: instantiating 'ConnectionPool(Redis)#connection()'

      @pool.connection do |conn|
            ^~~~~~~~~~

in ./src/sidekiq/pool.cr:65: instantiating 'ConnectionPool(Redis)#connection()'

      @pool.connection do |conn|
            ^~~~~~~~~~

in ./src/sidekiq/client.cr:135: instantiating 'Sidekiq::Pool#redis()'

      @ctx.pool.redis do |conn|
                ^~~~~

in ./src/sidekiq/client.cr:136: instantiating 'Redis#multi()'

        conn.multi do |multi|
             ^~~~~

in ./src/sidekiq/client.cr:136: instantiating 'Redis#multi()'

        conn.multi do |multi|
             ^~~~~

in ./src/sidekiq/client.cr:137: instantiating 'atomic_push(Redis::TransactionApi, Array(Sidekiq::Job+))'

          atomic_push(multi, payloads)
          ^~~~~~~~~~~

in ./src/sidekiq/client.cr:146: instantiating 'Array(Sidekiq::Job+)#each()'

        payloads.each do |hash|
                 ^~~~

in /usr/local/Cellar/crystal-lang/0.17.4/src/array.cr:813: instantiating 'each_index()'

    each_index do |i|
    ^~~~~~~~~~

in /usr/local/Cellar/crystal-lang/0.17.4/src/array.cr:813: instantiating 'each_index()'

    each_index do |i|
    ^~~~~~~~~~

in ./src/sidekiq/client.cr:146: instantiating 'Array(Sidekiq::Job+)#each()'

        payloads.each do |hash|
                 ^~~~

in ./src/sidekiq/client.cr:149: instantiating 'Sidekiq::Job+#to_json()'

          all << hash.to_json
                      ^~~~~~~

in ./src/sidekiq/job.cr:68: instantiating 'to_h()'

      to_h.to_json
      ^~~~

in ./src/sidekiq/job.cr:60: no overload matches 'Hash(String, JSON::Type)#[]=' with types String, (Array(Redis::RedisValue) | Bool | Int64 | Nil)
Overloads are:
 - Hash(K, V)#[]=(key : K, value : V)
Couldn't find overloads for these types:
 - Hash(String, JSON::Type)#[]=(key : String, value : Array(Redis::RedisValue))

      h["retry"] = retry if retry
       ^

Sidekiq breaks with kemal-session shard.

shards update throws the following error.
I might lack sidekiq version on shard. Only branch provided.

![image](https://user-images.githubusercontent.com/6760672/84786946-01a7c380-afb3-11ea-8886-8764d506a543.png

Specify number of workers

I have jobs which are very resource intensive (both memory and cpu). They are calling python scripts which process environmental data using python - scipy and numpy.

I would like to limit the number of workers that spin up (in my case to 4), and then run sidekiq workers on many separate servers rather than one large server.

Right now I see the message "Starting processing with 25 workers", which I am guessing comes from here. Would it be possible to check for the existence of an environment variable to use as a priority instead?

Idea: Extract Sidekiq Web to a separate shard

Sidekiq Web UI depends on Kemal. However, in Crystal community there are other web frameworks, such as Lucky or Amber. It would be cool to be able to use Sidekiq in them. Currently, fortunately, it's possible. But in the past I used to stumble upon errors coming from version mismatches between Kemal dependencies and Amber dependencies, leading me to be unable to do shards install.

Since I don't need Web UI in my Amber/Lucky project (if any, Sidekiq Web is run and deployed separately), it would be good to get rid of those dependencies and easiest way is to extract Sidekiq Web from main sidekiq.cr repository. I had to do it before on my fork and it's not that hard.

What do you think about it?

Middleware chain invoke not working in 0.18

This worked in 0.17.4. The code in question is here. What I've found is that the middleware chain is no longer returning a truthy/falsy value but always Nil.

Note this code is probably suboptimal, it's ported from Ruby. I'm realizing that I don't have a sufficient test suite for middleware and need to build test cases for the things middleware should be able to do.

      def invoke(job, ctx, &block)
        chain = entries.map { |k| k }
        next_link(chain, job, ctx, &block)
      end

      def next_link(chain, job, ctx, &block)
        if chain.empty?
          block.call
        else
          chain.shift.call(job, ctx) do
            next_link(chain, job, ctx, &block)
          end
        end
      end

If a chain has two middlewares, the call sequence should look like:

invoke
  next_link
    call middleware 1
      next_link
        call middleware 2
          next_link
            call invoke's block

Each middleware should be able to stop the execution early by returning Falsy (bug 1). Otherwise the entire chain should return the result of invoke's block (bug 2). Here's how the entire sequence is invoked.

Getting started with some web framework

I'm having some issues trying to figure out how to boot this version along with my Lucky app. I've used the Ruby version, so I'm familiar with how that all works.

It looks like I need this as a minimum:

require "sidekiq/cli"

cli = Sidekiq::CLI.new
server = cli.configure do |config|
end

Without that, calling any worker will throw an exception Sidekiq client has not been configured yet (Exception) even if I have the ENV vars setup.

But with that config, I now want to boot the sidekiq process. If I add the cli.run(server) in this file, then boot my app with crystal src/server.cr, it hits the sidekiq config and boots the server, then stops the lucky server from booting. I obviously need a process manager for this to work.

I tried adding a target to my shard.yml

targets:
  my_app:
    main: src/my_app.cr
  sidekiq:
    main: src/sidekiq.cr

Then I run shards build and end up getting some ld: library not found for -lcrt0.o (this usually means you need to install the development package for libcrt0.o) error.

Finally I decide to just putting worker: crystal src/sidekiq.cr in to my Procfile where I can just run lucky dev and have it boot. This fails though with the error You must register one or more workers to execute jobs!. So I have to require my workers specifically. So I add require "./workers/*" to the file, and now I get a compilation error because it doesn't know about any of the Lucky stuff.

I guess I could require all of that stuff twice, but at this point I feel like I'm not understanding the process and it's more just band-aids.

Rework JSON payload impl

Currently the JSON payload defined in Sidekiq::Job is static and cannot be extended. Rework the impl based on the new JSON.mapping capabilities wrt the finished macro so that applications and 3rd party middleware can introduce new fields into the job payload to be parsed automatically. This is one of the major failings that keeps Sidekiq.cr from working as well as Sidekiq.rb.

crystal-lang/crystal#3612
https://github.com/mperham/sidekiq.cr/blob/master/src/sidekiq/job.cr#L25

Test Mode

Is there any sort of test mode like in the ruby version? Or any advice on how to best work with sidekiq in the context of testing?

Thanks!