statianzo / que-web

A web interface for the Que queue

que-web

que-web is a web UI to the Que job queue.

Installation

Add this line to your application's Gemfile:

gem 'que-web'

And then execute:

$ bundle

Or install it yourself as:

$ gem install que-web

Usage

With config.ru

Add in config.ru:

require "que/web"

map "/que" do
  run Que::Web
end

Rails

In config/routes.rb:

require "que/web"
mount Que::Web => "/que"

Authentication

Devise

# config/routes.rb
authenticate :user do
  mount Que::Web, at: 'que'
end

Basic HTTP auth

In config/initializers/queweb.rb:

Que::Web.use(Rack::Auth::Basic) do |user, password|
  [user, password] == [ENV["QUEWEB_USERNAME"], ENV["QUEWEB_PASSWORD"]]
end

Then add the two environment variables to your production environment.

Docker

Run:

docker run -e DATABASE_URL=postgres://username:password@hostname/db_name -p 3002:8080 joevandyk/que-web

Or use docker/Dockerfile to build your own container.

que-web's Issues

Returns part of HTML

For some reason, it returns only a part of HTML page:

• on 'running' page I see a withe page, and it returns only a part of HTML (only part of the header, without the body)
• on 'scheduled' page I see only 2 jobs from 7, and I can't see any actions buttons at the bottom of the page
• on 'failing' page I see only 6 jobs from 289 without paginations and actions buttons at the bottom of the page

Gems:

• locks (0.2.0)
• que-scheduler (3.3.0)
• que-web (0.9.3)
• que (1.0.0.beta4)

Works great on the local machine but have an issue on Heroku production server

Could you please help to understand where can be the issue

release 0.9.2

@statianzo Could you release 0.9.2 with 2 small bugfixes https://github.com/statianzo/que-web/blob/master/CHANGELOG.md#092---unreleased

Eventually you could add me as a gem owner, so I can also release it if you don't mind.
My email at rubygems is "[email protected]" https://rubygems.org/profiles/morgoth

Feature suggestion: Add an overdue jobs section

Currently, there are three main sections in Que web - running, scheduled and failing.

If you have jobs enqueued in the future (eg run_at: 1.day.from_now) it makes it harder to see jobs that are overdue/waiting to run (eg because of a backlog in the workers) because they are in the same bucket (in Que web) as jobs that we don't really care about yet.

It would be very handy to split out those jobs that can/should run immediately (ie run_at <= now() from those that are scheduled for some time later.

PS. Thanks for you efforts in making this useful Que add-on.

Can't actually retry expired jobs

This isn't a que-web bug per se, but it is a bit confusing. If a job's error_count has exceeded max_retry_count and expired_at has a value, this job is effectively "dead" and yet que-web allows you to retry it, and gives the impression that it has been retried (run_at updates) - yet the job is not actually re-run. I don't know if there's a way around this, though I've opened an issue with que to find out - if so, we should fix in que-web, otherwise add some logic to disable the retry button for such jobs.

Support Multitenanted schemas

Que inserts and reads jobs in the public schema by default (see this example in the source), but que-web does not mention the schema in queries. This conflicts with multi-tenancy rack middleware like apartment.

Would it be problematic to specify the DB schema in all SQL queries?

Show processed jobs

It would be great to be able to see the number of jobs and inspect the particular jobs which have processed successfully on the webpage.

Bulk delete and run immediately

Right now if you want to remove 5 jobs, you have to click 5 trash cans. We should be able to select jobs via checkbox and perform an action.

Support queues

I would love to be able to group jobs by the queue, similar to how resque's interface works. So you pick from the list of queues and then you see the set of jobs for that queue.

Release new version

@statianzo Would it be possible to release new gem version with latest changes?

Templates rendering as strings not markup

Removal of erubis in this commit 25962a2 caused regression in template rendering.

Still need double equals.

Show job's kwargs column (for Ruby 3 / Que 2 jobs)

I haven't used this gem yet, but when the PR to add Que 2 support landed, I thought the diff was suspiciously small. I see that args is listed in the list jobs and show job views, but kwargs is not. kwargs is a new column added in Que 2 for Ruby 3 support.

Deleting or rescheduling a job locks all enqueued jobs.

Looks like this regressed in #30.

Long arguments in job lists overflow

Right now long job arguments will overflow off the side of the page. Let's wrap them and potentially truncate them.

Search sanitisation incorrectly removes `Li` from `LinkQueJob`

With Using LinkQueJob as the search param - the search text box is filled with nkQueJob

This looks like a typo in the regex I changed here - the /.../i is written as /...i/

Receiving "undefined method 'utc' error" when attempting to transition to the 'failing' tab

Error message in terminal:

2015-04-08 10:04:28 - NoMethodError - undefined method utc' for "2015-04-08 17:08:06.843141":String: /...gems/que-web-0.3.2/lib/que/web.rb:112:inrelative_time'

Show truncated errors in Failing list

To help get an idea of what caused a failure, showing a truncated error message would save from having to open each job.

Flash message upon actions

When a job is scheduled or run immediately, a flash message should be displayed indicating that the action has occurred.

Version and date clip pagination buttons

The poorly implemented footer makes pagination difficult because it sits on top of the "Next" and "Previous" buttons.

Malicious job args are XSS vulnerable

Job args with <script> in them are not escaped properly at the moment.

Missing `present?` method

The project seems to require an implementaiton of .present? - See here: https://github.com/statianzo/que-web/blob/v0.7.1/lib/que/web.rb#L154

I can get que-web it to work when I add the following line to my Gemfile:

gem 'activesupport', require: 'active_support/core_ext/object/blank'

Am I missing something, like a different library that provides the .present? implementation?

Allow config to specify some jobs as self enqueueing

We have a number of self-enqueueing jobs that we never want to delete, as they run forever.

We would like to specify that que-web should not delete these if a "Delete All" was pressed, or show an extra warning that that is about to happen if they are included. Also, the "single delete" (dustbin) image should be hidden or show an extra warning.

Additionally, a warning could be shown if they are missing.

For example, jobs like que-scheduler should always be present.

Would you be open to a PR for this?

Release a new version

Would it be possible to release a new version of the Gem on rubygems.
There are some good new features such as search which would be great to have.

Thanks!

Precompile assets

Is there a way to precomile the assets used by que-web (jQuery, Font Awesome, etc.), perhaps to a specific path?

I'm having a hard time getting que-web to load its assets from the right place when running my app in Docker at a relative path (not at the root) in a cluster that uses HAProxy to do the routing. That is, at my.cluster.com/my-app. Works fine when running at a relative root locally, and at a relative path in Docker on my machine.

Rescheduled job is not worked if there more than one jobs present when rescheduling

If there are >1 jobs in the que_jobs table (and at least one has failed), rescheduling a failed job will make it show in "running jobs" but it will never actually run (unless the que-web process exits).

This is a very similar issue to #31 - but whereas there, we were locking all jobs, we now lock a single job N times (once per row in the jobs table). As per the documentation,

A lock can be acquired multiple times by its owning process; for each completed lock request there must be a corresponding unlock request before the lock is actually released.

Unfortunately, we only issue a single unlock request, meaning that, unless the failed job was the only job, we will not issue enough unlock requests and the job will remain locked by que-web (which has the effect of making it show in the Running section).

This issue, and #31 and #30 have highlighted the difficulty in trying to construct a (correct) atomic SQL statement to be executed by Postgres to lock/modify/unlock a single job. Many apologies for the bugs that we've introduced, they've highlighted some behaviour that is quite subtle! I think the lack of tests for the tricky behaviour hasn't helped, but we could have also done a better job of deeply understanding the queries we were writing.

I have a fix that will abandon the atomic query approach, and instead use separate DB queries to lock, then modify, then unlock a given job, which is simpler and easier to reason about, for a negligible reduction in performance.

Add documentation on passwording the /que endpoint

# config/initializers/queweb.rb
Que::Web.use(Rack::Auth::Basic) do |user, password|
  [user, password] == [ENV["QUEWEB_USERNAME"], ENV["QUEWEB_PASSWORD"]]
end
``
Then add the two environment variables to your production environment

Pagination controls are getting escaped

Pagination controls are getting rendered as escaped html. This means no paging :-(

undefined method utc for string

Dependencies

PgSQL 12
ruby: 2.6.5
rails: 5.2.3
pg: 1.1.4
que: 1.0.0.beta4 (commit 065981)
que-web: HEAD (commit 3f7b7b7)

Reproduce Condition

Visit /que/scheduled

How System Behaves

• System displays the message:

NoMethodError at /que/scheduled
undefined method `utc' for "2020-09-06 22:17:03":String
in que-web/lib/que/web.rb in relative_time:188
que-web/web/views/scheduled.erb:26

My best guess so far is que-web/lib/que/web.rb:39

scheduled_jobs = Que.execute SQL[:scheduled_jobs], [pager.page_size, pager.offset, search]

returns run_at as a string, example: "2020-09-06 22:17:03"

The relative_time helper expects a Time object at lib/que/web.rb:189

      def relative_time(time)
        %{<time class="timeago" datetime="#{time.utc.iso8601}">#{time.utc}</time>}
      end

and the String class is missing .utc

Proposed Solution

https://github.com/AlanLattimore/que-web/tree/undefined-method-utc-for-string

      def relative_time(time)
        time = Time.parse(time) if time.is_a?(String)
        %{<time class="timeago" datetime="#{time.utc.iso8601}">#{time.utc}</time>}
      end

based on que-rb/que@87d40e7

Support for Que 1.0

Hi there and thanks for all the work on que-web!

Que 1.0 beta releases have been coming out steadily, but they are not backwards compatible with 0.* branch.

Are there any plans for que-web to support the new versions?

Clarification request => Devise code goes where?

It would be nice if the readme was clearer as to where the Devise code needs to be added. I'm guessing it's the Devise initializer.

Problem with rails 5.0.0.rc1 : must use Sinatra master branch

Adding gem 'que-web' to a Rails 5.0.0.rc1 with ruby 2.3.1 make rails crash at startup:

/home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require': cannot load such file -- rack/showexceptions (LoadError)
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `block in require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:259:in `load_dependency'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/sinatra-1.0/lib/sinatra/showexceptions.rb:1:in `<top (required)>'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `block in require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:259:in `load_dependency'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/sinatra-1.0/lib/sinatra/base.rb:6:in `<top (required)>'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `block in require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:259:in `load_dependency'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/sinatra-1.0/lib/sinatra.rb:4:in `<top (required)>'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `block in require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:259:in `load_dependency'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/activesupport-5.0.0.rc1/lib/active_support/dependencies.rb:293:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/que-web-0.4.0/lib/que/web.rb:1:in `<top (required)>'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler/runtime.rb:91:in `require'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler/runtime.rb:91:in `rescue in block in require'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler/runtime.rb:68:in `block in require'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler/runtime.rb:61:in `each'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler/runtime.rb:61:in `require'
  from /home/username/.rvm/gems/ruby-2.3.1/gems/bundler-1.11.2/lib/bundler.rb:99:in `require'
  from /home/username/project/ex/config/application.rb:7:in `<top (required)>'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:88:in `require'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:88:in `block in server'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:85:in `tap'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:85:in `server'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands/commands_tasks.rb:49:in `run_command!'
  from /home/username/project/ex/.bundle/ruby/2.3.0/gems/railties-5.0.0.rc1/lib/rails/commands.rb:18:in `<top (required)>'
  from bin/rails:4:in `require'
  from bin/rails:4:in `<main>'

I think this is related to this Sinatra issue: sinatra/sinatra#1055 and that one should use the master branch to support rails >= 5.0.0

The documentation may need to be updated.

Thanks,

Failing jobs "Delete all" does not respect search

Using a search query on the failing jobs page correctly restricts the displayed jobs. Upon pressing "Delete All" the prompt asks "Delete all N jobs?" where N is the count of jobs restricted by the search, however, upon confirming the deletion all failing jobs are deleted, whether matched by the search query, or not.

As an example, with the following jobs (which are failing as I have not created the corresponding classes):

postgres@localhost:backend> SELECT job_class, error_count FROM que_jobs
+-------------+---------------+
| job_class   | error_count   |
|-------------+---------------|
| FooQueJob   | 4             |
| FooQueJob   | 4             |
| BarQueJob   | 4             |
+-------------+---------------+

the failed job page shows:

and if we search for "Foo" it shows:

if we press "Delete All" we are asked:

pressing "Ok" we see (after clearing the search)

and indeed, all the jobs have been deleted, not just those matching "Foo":

postgres@localhost:backend> SELECT job_class, error_count FROM que_jobs
+-------------+---------------+
| job_class   | error_count   |
|-------------+---------------|
+-------------+---------------+

N.b. we are pinned to que-web 0.7.1, since we have not yet upgraded to Que 1.X

Error when using search on Running tab

If a search is entered on the Running tab, I get the following error with que-web 0.7.1:

NoMethodError - undefined method `job_class' for #<ActiveSupport::HashWithIndifferentAccess:0x00007f4eaa30f160>:

This is because this line calls a job_class method on the entries of Que.worker_states which are Hashes and thus don't respond to that method

Update gemspec to allow newest Que

statianzo there is Que 1.3.1 already, could you update gemspec to reflect that?

Job rerun from que-web will appear to run but never complete in 0.6.0

My previous change (#28) attempted to try and obtain an advisory lock before updating a job to prevent duplication of already-running jobs.

It was successful in preventing updates to jobs that were already running (i.e. when the lock couldn't be obtained), but unfortunately, if que-web does obtain the lock (and therefore updates the job) the job will never be able to be worked, as que-web doesn't release the lock (at least until the DB session exits).

The effect of this is that the job shows as "running" in que-web (as a job being locked is enough for que-web to treat it as running), but the job is not actually running, and never will (until the lock is released by que-web).

The solution will of course to be to explicitly release the lock.

Question: How do you make this work?

Thank you for this nice gem, very useful!

I just have a small doubt, Im trying to replicate this

que-web/lib/que/web/sql.rb

Line 95 in d663559

delete_job: delete_jobs_query(lock_job_sql),

Mapping a function inside a hash so SQL[:delete_job] calls the method, when I try to use a function inside a hash in plain ruby I get undefined method delete_jobs_query'` error...

Now sure how are you making this work, any explanation?

Thank you!

PD: I think this is a StackOverflow question but did not find a relevant tag for this gem so better to ask directly.

Not compatible with gem que-2.2

I have updated my que gem from que-1.4 to gem que-2.2 and seems like this que-web is not compatible with that

Here are my error logs

Bundler could not find compatible versions for gem "que":
  In Gemfile:
    que (~> 2.2)

    que-web (~> 0.9.4) was resolved to 0.9.4, which depends on
      que (~> 1)

Support pending jobs

Might be nice to have a "Pending Jobs" section that shows jobs with a run_at in the past. Scheduled jobs would show jobs with a run_at in the future.

Confirmations on delete and "run immediately"

Accidentally touching these can have adverse effects. We should require a confirmation for performing these actions.

UndefinedColumn at /running

I can not access the /running route with Que schema version 3.

I get an error saying the column pg_waiting_on_lock does not exist.

I believe this is an issue for everyone using the newer version of Que.

Relative dates

Instead of ugly UTC dates, show "3 hours from now" on the scheduled and failing views.

SyntaxError at /que/

After upgrading my app to Rails 5, I noticed that I could no longer access the /que endpoint (using current master of Sinatra). I fixed this by using the latest version instead (2.0.0). The error has to do with Sinatra being unable to read raw ERB tags (<%== %>) but I'm not entirely sure of the cause. Either way, should probably update the readme to advise the latest version of Sinatra in Gemfiles.

lib/ruby/gems/2.4.0/gems/que-web-0.6.3/web/views/index.erb:6: syntax error, unexpected '=' at " "; @_out_buf.concat((= erb :_search ).to_s); @_out_b ^