gavinwahl / django-postgres-queue Goto Github PK

• Tasks should be logged
• Should use python logging with no funny business like celery
• Exceptions in tasks should show a stacktrace but not crash the worker

Hi there,
Is there a way to find following things from queue:

1. Oldest message from queue?
2. Queue Size

this seems very cool, I'm going to use it

in rabbitmq, amqp, kafka etc the name of a queue is an important design capability

for example, you might have a "real time queue" and a "batch queue" with different number of workers allocated

or, a "first time queue" and a "retry queue" or "failed queue"

or just, in general, two django apps or parts of a large system that both want to use a postgres-queue without getting crosstalk among their messages

unless I misunderstand, there isn't an equivalent here?

it seems like it could be added by having an indexed queue_name field on dpq.models.Job, then having the dequeue method accept the name and use it as part of the where clause.

for backwards compatibility, None or some default name could be used; but in the future dpq.queue.Queue could have a name field that it attaches to jobs before enqueueing

another design approach might be to actually have separate tables for each logical queue name; not sure if there are advantages to table-level isolation here or not

• Worker/Queue implentation boilerplate
• Comparison to celery (no funny business -- automatically importing your code, importing code by string, autodiscovery)
• mention not to use transaction.on_commit
• systemd unit file for worker
• how to log to a file
• Monitoring queries/pgrowlocks
• sentry handler
• How to run tasks in tests

I know celery discourages use of always eager. But I love it, and I always will.

I have written my own version of always eager internally, but it might be nice to add to the library? It is useful during end-to-end testing where you definitely want the thing you are queuing to run during the test.

You have multiple queues created. e.g.

queue1 = AtLeastOnceQueue(tasks={'task1': task1})
queue2 = AtLeastOnceQueue(tasks={'task2': task2})

Once woken, workers will work through all the tasks on the main job table.

But this worker's queue may not have a task which matches one in its available task list. You'll get a KeyError here:
https://github.com/gavinwahl/django-postgres-queue/blob/master/dpq/queue.py#L23

Instead of raising KeyError, the worker should do something useful. e.g.

• Ignore the task, it obviously was meant for someone else
• Not fetch that task in the first place, since it cannot complete it
• Raise a specific error that could be caught e.g. DpqInvalidTask

I will make a PR with a test case highlighting the problem, and I think option 1 or 2 above is the best, i'll implement one of them.

WARNINGS:
dpq.Job.args: (fields.W904) django.contrib.postgres.fields.JSONField is deprecated. Support for it (except in historical migrations) will be removed in Django 4.0.
        HINT: Use django.db.models.JSONField instead.

I think this should be a pretty straightforward change? I'm not super familiar with python library, but if you don't have time to work on this, I may be able to send a PR.

Thanks for this lib!

Hi

i like this module, and i want to migrate my application to use it.

my current setup is that i am using supervisord to run workers for different tasks
and each such worker has parameter --number-of-threads, --number-of-processes
this parameter tells how many threads/processes each worker will run in parallel

is there ability in this module to run multiple workers at parallel?
probably i need to create some kind of WorkerPool class and run call_command('worker')
inside it in multiple processes or is there some natural way in this module how to run multiple workers?

thanks for any answer