Concurrency Blocked Jobs do not respect queue priority about solid_queue

[pasl]

Hi @rosa,

Thank you for your help.

I was under the impression that if we put "*" inside a worker's queue configuration, it means that this worker will pick up jobs from any queues.

My goal with this configuration was to ensure all jobs sent to any queue other than real_time, background, or best_effort, like default for example, would be picked up by the worker and executed with middle priority.

Actually, this configuration works as expected when I am not using the limits_concurrency feature. All jobs are executed in the order of the queue position from real_time to * (anything else) to background, and then best_effort.

[edit]
Not true... You were right. After further testing with Jobs without concurrency, Queues are not prioritize when "" is present. Without the "" queues are now working as expected for normal Jobs.
[/edit]

However, when I add the limits_concurrency feature, the queue position is not being considered, and jobs are executed in their job ID order.

New test (with priority):

# Schedule many jobs with priority 10 inside the background queue
1000.times do
  GetPredictionJob.set(queue: :background, priority: 10).perform_later(...)
end

# Schedule 1 job with priority 1 inside the background queue
GetPredictionJob.set(queue: :background, priority: 1).perform_later(...)

This time, the latest job passed in front of the others.

So, I think that once the job has been placed into the blocked job because of limits_concurrency, the job's priority is being kept, but not the queue order.

[rosa]

Hey @pasl, thanks for opening this issue and trying out Solid Queue! I think the concurrency controls don't play a role here, I think what happens is that your worker is set to run all queues, besides background and best_effort, as "*" is included. If "*" is included, your configuration behaves in the same way as:

workers:
    - queues: "*"

So the order followed is first numeric priority (which in your case it's irrelevant because you're using queue-based priority) and then job_id, so the jobs you enqueued first get executed first.

For the order you want, I think you could try with this configuration instead:

workers: 
  - queues: [ real_time, background, best_effort ]

[rosa]

Thanks for checking this, @pasl, and sorry for the delay. Let me try to understand the issue now.

So, I think that once the job has been placed into the blocked job because of limits_concurrency, the job's priority is being kept, but not the queue order.

Does this still happen when you're using

- queues: [ real_time, background, best_effort ]

as configuration?

[pasl]

Hi, correct.

My configuration is slightly different, but the * is removed.

 - queues: [ real_time, default, background, best_effort ]

And yes, from what I can see, once the jobs have been sent to the blocked list because they have been limited by the concurrency rules, queues don't seem to matter in terms of priority. However, job priority will be honored.

[rosa]

Oh, I see the problem now, yes. This is correct; only priority is honored when unblocking jobs, because picking the next job to be unblocked only takes priority into account, not queue. The queue can't be taken into account because the job doesn't know the worker's order or queue priority order, it just knows about its own queue and concurrency key, and can only pick a job to unblock based on its concurrency key. Then, once the jobs are unblocked, they'll be picked from the queue in queue priority order, but what happens in your case is that jobs aren't unblocked in that order, so the effect in the end is that queue doesn't matter.

I don't think we can change this without completely changing how concurrency controls work, so I'll document this instead.