saurabhnanda / odd-jobs Goto Github PK
View Code? Open in Web Editor NEWHaskell job queue with admin UI and loads of other features.
Home Page: https://www.haskelltutorials.com/odd-jobs/
License: BSD 3-Clause "New" or "Revised" License
Haskell job queue with admin UI and loads of other features.
Home Page: https://www.haskelltutorials.com/odd-jobs/
License: BSD 3-Clause "New" or "Revised" License
I have tried to use the error handler like following
mkConfig
logger
jobTableName
dbPool
(MaxConcurrentJobs 50)
jobRunner
( \c -> do
c {cfgDefaultMaxAttempts = 1,
cfgOnJobFailed = [JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"]
}
)
It throws the following error
_ Couldn't match type _a1_ with _()_
_a1_ is a rigid type variable bound by
a type expected by the context:
forall a1. [JobErrHandler a1]
at app/Main.hs:61:41-109
Expected type: JobErrHandler a1
Actual type: JobErrHandler ()
_ In the expression:
JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"
In the _cfgOnJobFailed_ field of a record
In a stmt of a 'do' block:
c {cfgDefaultMaxAttempts = 1,
cfgOnJobFailed = [JobErrHandler
$ \ (e :: SomeException) job failMode -> print "Test"]}
|
61 | cfgOnJobFailed = [JobErrHandler $ \ (e :: SomeException) job failMode -> print "Test"]
Could anyone please help in resolving this. Thanks in advance.
I have a use case for which Odd Jobs is very appealing, but it requires the ability to handle somewhat larger job payloads. Right now, creating a job with a large payload fails in the database with this error:
ERROR: payload string too long
CONTEXT: SQL statement "SELECT pg_notify('job_created_jobs', row_to_json(new)::text)"
PL/pgSQL function notify_job_monitor_for_jobs() line 2 at PERFORM
Out of the box, PostgreSQL limits the size of the notification payload to 8000 bytes (see https://www.postgresql.org/docs/12/sql-notify.html). Because the trigger created by createNotificationTrigger
passes the full job table row (serialized to JSON) to pg_notify
, the maximum size of the job payload is significantly less that 8000 bytes.
Fortunately, the de-serialization of the notification payload in jobEventListener
only cares about the id
, run_at
, and locked_at
columns. It seems that it should be easy to accommodate larger job payloads by changing the trigger to only pass these values, suitably JSON-serialized, to pg_notify
, instead of the entire row.
Where the trigger function currently has row_to_json(new)::text
, simply substituting json_build_object('id', new.id, 'run_at', new.run_at, 'locked_at', new.locked_at)::text
seems to do the trick.
Is there a problem that I'm missing with in this approach?
I am interested in having a priority attached to jobs. That would allow me to add a large number of relatively unimportant jobs to the queue without having to worry about them preventing more important jobs from running.
When createJobTable is called with a table name which contains a dot for example myschema.jobs
, it fails with following error
sqlErrorMsg = "syntax error at or near \".\""
The index creation seems to fail.
create index idx_myschema.jobs_created_at on one.jobs_test3(created_at)
The last version available on Hackage, 0.2.2
, is over 12 months old. Since then, there have been some minor useful fixes - but also a major fix, namely #43 , which solves a Postgres problem that is very easy to run into, and which I have hit with large job payloads.
It would be much appreciated if the recent fixes could be included in a new Hackage release, so developers like me don't trip up on old issues.
This might sound kind of silly, but without it ResourceT based connection pools put the connection back too early.
In the real world this translates to:
If you use Persistent and postgresql with odd-jobs with a connection pool you get "libpq: failed another command already in progress" errors.
We had this issue in production and it debugging it led me to:
A more concise example you can play with yourself is at https://gist.github.com/codygman/d38a049092301ade8e8e1bb362cac778#file-main-hs-L23
I also created an issue on Persistent but I'm starting to think Persistent might not be able to fix this if it requires wrapping calls with Async:
The config has a few callbacks for job lifecycle events:
cfgOnJobStart
cfgOnJobSuccess
cfgOnJobFailed
cfgOnJobTimeout
Of those, cfgOnJobSuccess
and cfgOnJobFailed
appear to be called correctly. As far as I can tell the other two, cfgOnJobStart
and cfgOnJobTimeout
, are never used.
cfgOnJobStart
is wired into the onJobStart
method of the HasJobRunner
class for RunnerM
, but onJobStart
itself is never called.
cfgOnJobTimeout
appears to be completely unused.
I'm looking into using odd-jobs to replace a home-spun job queue implementation. The web UI is really useful, but I would like to put it behind a subpath on an auth-protected domain which I use for all my dev UIs, so that I can still access it remotely.
Currently, I using a reverse-proxy to do that - so the web UI is served on a URL like
https://my.dev.domain/jobs/
and that routes to
http://127.0.0.1:<odd-jobs-port>
However, the UI doesn't currently really support this. All of the static assets and links are broken, since they expect to be served from the URL root /
, rather than the /jobs/
path. It would be great if odd-jobs would support custom URL roots, so I could use it in the above way.
Using CREATE TABLE IF NOT EXISTS
here
odd-jobs/src/OddJobs/Migrations.hs
Line 13 in 0236697
CREATE TABLE
will stop this function from failing if the table already exists.JobIds are currently Int
which is potentially limited to 2^29. They should be Int64 at least.
Our team is giving odd-jobs a try and we're loving it! The only current concern we have is that we would like to avoid exponential backoff for failures since we only deal with transient failures. Would it be possible to add a config to modify it, such as adding a limit to the time added, or removing it altogether?
This is the line we are concerned with:
Line 408 in b9649f5
I could open a PR to add a config but looks like the roadmap is already planning on doing something with failures per job don't want to step on toes.
Problem context and a hack-solution
I have a situation where the jobs need to be limited by a resource of some type. For example, a resource should not have more than K jobs running at a time. Resources could be idenified by a text column.
It appears the best way forward is to schedule all jobs, keep my own table for resource consumption, and reject the excess jobs by re-queuing with a status of Retry
. This enforces the resource limit and makes the excess job(s) continually re-scheduled on some interval (depending on jobRunAt). Even then, it isn't clear how to set a job to be marked for Retry... perhaps just a direct call to saveJobIO
.
Proposing a better solution
We could solve this and, to the high level user, we would only introduce two fields:
jobResource :: Maybe Text
column describing the resource consumed by this job.jobMaxResourceCount :: Maybe Int
column describing the max resource consumption.The listener and poller would have to include another filter. Something like SELECT * FROM $table outer WHERE jobMaxResourceCount > ( SELECT COUNT(*) FROM $table jobResource = outer.jobResource )
.
Thoughts? It would be great to hear where you're planning on taking this library.
Other Thoughts
Odd jobs looks wonderful, thanks for open-sourcing it. A few aspects feel framework-ish (i.e. it gets to feeling prescriptive such as on the logging) and that makes me wonder how hard it might be to rework my queue to use odd jobs. That said, the point in the design space is spot on as well as the exposed level of abstraction. Much appreciated!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.