saurabhnanda / odd-jobs

Haskell job queue with admin UI and loads of other features.

Introduction

• Odd Jobs home page - contains a description of top-level features of this library
• Getting started & implementation guide for Odd Jobs
• Haskell Job Queues: An Ultimate Guide - A detailed writeup on why we built Odd Jobs and how it compares against other similar libraries.
• Start reading Hackage documentation from OddJobs.Job (Note: Please ensure you're reading docs for the correct version of the library)
• Open an issue on Odd Jobs Github repo if you need help, or want to collaborate.
• Please ⭐ the repository for good karma and ❤️

odd-jobs in production?

If you are already using, or considering using, odd-jobs in production, please read production usage-reports. It would be great if you could add your own usage-report to that discussion thread as well. We need more success stories of Haskell in production!

Contributing

Please read the contribution guidelines

Development

Prerequisites

• The Haskell Tool Stack
• libpq-dev library (required for PostgreSQL dependency)

Build

stack build

Running tests

Add the users to postgresql:

CREATE USER jobs_test WITH SUPERUSER PASSWORD 'jobs_test';
CREATE DATABASE jobs_test OWNER jobs_test;

odd-jobs's Issues

Production usage reports

Need help in using JobErrHandler

I have tried to use the error handler like following

mkConfig
                logger
                jobTableName
                dbPool
                (MaxConcurrentJobs 50)
                jobRunner
                ( \c -> do
                    c {cfgDefaultMaxAttempts = 1,
                       cfgOnJobFailed = [JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"]
                      }
                )

It throws the following error

    _ Couldn't match type _a1_ with _()_
      _a1_ is a rigid type variable bound by
        a type expected by the context:
          forall a1. [JobErrHandler a1]
        at app/Main.hs:61:41-109
      Expected type: JobErrHandler a1
        Actual type: JobErrHandler ()
    _ In the expression:
        JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"
      In the _cfgOnJobFailed_ field of a record
      In a stmt of a 'do' block:
        c {cfgDefaultMaxAttempts = 1,
           cfgOnJobFailed = [JobErrHandler
                               $ \ (e :: SomeException) job failMode -> print "Test"]}
   |
61 |                        cfgOnJobFailed = [JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"]

Could anyone please help in resolving this. Thanks in advance.

Complete the code for a simple example

JobId should be Int64

JobIds are currently Int which is potentially limited to 2^29. They should be Int64 at least.

Setup CI using Github Actions for odd-job (job-queue)

createJobTable fails when the table name has a dot

When createJobTable is called with a table name which contains a dot for example myschema.jobs, it fails with following error

sqlErrorMsg = "syntax error at or near \".\""

The index creation seems to fail.

create index idx_myschema.jobs_created_at on one.jobs_test3(created_at)

Tutorial > DB Connection Pool

Integrate odd-jobs (job-queue) with Snap

Proper test suite

Implement reaper for failed/stale jobs

More structured logging

Some job callbacks are not used

The config has a few callbacks for job lifecycle events:

• cfgOnJobStart
• cfgOnJobSuccess
• cfgOnJobFailed
• cfgOnJobTimeout

Of those, cfgOnJobSuccess and cfgOnJobFailed appear to be called correctly. As far as I can tell the other two, cfgOnJobStart and cfgOnJobTimeout, are never used.

cfgOnJobStart is wired into the onJobStart method of the HasJobRunner class for RunnerM, but onJobStart itself is never called.

cfgOnJobTimeout appears to be completely unused.

Admin UI > Display count of jobs that match a given filter

Relase announcement

Template Haskell helpers for easier creation of job-runners

Add warnings to incomplete packages

Admin UI > Handle edge-case where jobTypeFilter=unknown

Building blocks for web UI

Use fill-factor in jobs table

Delete jobs upon successful completion

Tutorial > Deployment

Publish detailed tutorial and link it off from haddocks

Add priority to jobs?

I am interested in having a priority attached to jobs. That would allow me to add a large number of relatively unimportant jobs to the queue without having to worry about them preventing more important jobs from running.

Tutorial > Concurrency Control

Tutorial > Think about feature-complete example

Admin UI > Less/more toggle for "last error"

Integrate odd-jobs (job-queue) with Yesod

Building blocks for CLI

Release a new version

The last version available on Hackage, 0.2.2, is over 12 months old. Since then, there have been some minor useful fixes - but also a major fix, namely #43 , which solves a Postgres problem that is very easy to run into, and which I have hit with large job payloads.

It would be much appreciated if the recent fixes could be included in a new Hackage release, so developers like me don't trip up on old issues.

Complete the documentation and publish it

PostgreSQL notification payload limit should not restrict job payload size

I have a use case for which Odd Jobs is very appealing, but it requires the ability to handle somewhat larger job payloads. Right now, creating a job with a large payload fails in the database with this error:

ERROR:  payload string too long
CONTEXT:  SQL statement "SELECT pg_notify('job_created_jobs', row_to_json(new)::text)"
PL/pgSQL function notify_job_monitor_for_jobs() line 2 at PERFORM

Out of the box, PostgreSQL limits the size of the notification payload to 8000 bytes (see https://www.postgresql.org/docs/12/sql-notify.html). Because the trigger created by createNotificationTrigger passes the full job table row (serialized to JSON) to pg_notify, the maximum size of the job payload is significantly less that 8000 bytes.

Fortunately, the de-serialization of the notification payload in jobEventListener only cares about the id, run_at, and locked_at columns. It seems that it should be easy to accommodate larger job payloads by changing the trigger to only pass these values, suitably JSON-serialized, to pg_notify, instead of the entire row.

Where the trigger function currently has row_to_json(new)::text, simply substituting json_build_object('id', new.id, 'run_at', new.run_at, 'locked_at', new.locked_at)::text seems to do the trick.

Is there a problem that I'm missing with in this approach?

Tutorial > Logging

Use prepared-statements and bind-params for SQL statements where appropriate

Tutorial > Complete library comparison

Make defaultLockTimeout configurable at job-runner level

Resource-limited jobs

Problem context and a hack-solution

I have a situation where the jobs need to be limited by a resource of some type. For example, a resource should not have more than K jobs running at a time. Resources could be idenified by a text column.

It appears the best way forward is to schedule all jobs, keep my own table for resource consumption, and reject the excess jobs by re-queuing with a status of Retry. This enforces the resource limit and makes the excess job(s) continually re-scheduled on some interval (depending on jobRunAt). Even then, it isn't clear how to set a job to be marked for Retry... perhaps just a direct call to saveJobIO.

Proposing a better solution

We could solve this and, to the high level user, we would only introduce two fields:

• jobResource :: Maybe Text column describing the resource consumed by this job.
• jobMaxResourceCount :: Maybe Int column describing the max resource consumption.

The listener and poller would have to include another filter. Something like SELECT * FROM $table outer WHERE jobMaxResourceCount > ( SELECT COUNT(*) FROM $table jobResource = outer.jobResource ).

Thoughts? It would be great to hear where you're planning on taking this library.

Other Thoughts

Odd jobs looks wonderful, thanks for open-sourcing it. A few aspects feel framework-ish (i.e. it gets to feeling prescriptive such as on the logging) and that makes me wonder how hard it might be to rework my queue to use odd jobs. That said, the point in the design space is spot on as well as the exposed level of abstraction. Much appreciated!

Modifying exponential backoff for failures

Our team is giving odd-jobs a try and we're loving it! The only current concern we have is that we would like to avoid exponential backoff for failures since we only deal with transient failures. Would it be possible to add a config to modify it, such as adding a limit to the time added, or removing it altogether?

This is the line we are concerned with:

odd-jobs/src/OddJobs/Job.hs

Line 408 in b9649f5

, jobRunAt=(addUTCTime (fromIntegral $ (2::Int) ^ jobAttempts) t)

I could open a PR to add a config but looks like the roadmap is already planning on doing something with failures per job don't want to step on toes.

Odd jobs create table fails with exception if table already exists.

Using CREATE TABLE IF NOT EXISTS here

odd-jobs/src/OddJobs/Migrations.hs

Line 13 in 0236697

createJobTableQuery = "CREATE TABLE ?" <>

instead of CREATE TABLE will stop this function from failing if the table already exists.

Kill a running job

Implement cron-like repetitive job scheduling in odd-jobs

Support for custom URL roots

I'm looking into using odd-jobs to replace a home-spun job queue implementation. The web UI is really useful, but I would like to put it behind a subpath on an auth-protected domain which I use for all my dev UIs, so that I can still access it remotely.

Currently, I using a reverse-proxy to do that - so the web UI is served on a URL like

https://my.dev.domain/jobs/

and that routes to

http://127.0.0.1:<odd-jobs-port>

However, the UI doesn't currently really support this. All of the static assets and links are broken, since they expect to be served from the URL root /, rather than the /jobs/ path. It would be great if odd-jobs would support custom URL roots, so I could use it in the above way.

Fix type-signature of onJobFailed and onJobPermanentlyFailed

Make TableName the first argument in functions like saveJobIO, etc.

Admin UI > Filter by job-runner

Admin UI > Display "pills" for applied filters

Recommend users fork jobs with Async

This might sound kind of silly, but without it ResourceT based connection pools put the connection back too early.

In the real world this translates to:

If you use Persistent and postgresql with odd-jobs with a connection pool you get "libpq: failed another command already in progress" errors.

We had this issue in production and it debugging it led me to:

snoyberg/conduit#425

A more concise example you can play with yourself is at https://gist.github.com/codygman/d38a049092301ade8e8e1bb362cac778#file-main-hs-L23

I also created an issue on Persistent but I'm starting to think Persistent might not be able to fix this if it requires wrapping calls with Async:

yesodweb/persistent#1199

Complete v1 of web UI

Add odd-jobs (job-queue) to Stackage

Tutorial > Convert simple example to Literate Haskell

Tutorial > Remember to add index on job-type