seedofjoy / darq Goto Github PK

Add ability to optionally pass ctx to task, like this:

@task(with_ctx=True)
def foobar(ctx):
    log.info('Foobar try %s', ctx['job_try'])

Hi
Is there a way to increase graceful_shutdown_timeout from worker config?

Darq Worker can easily handle 10+ I/O-bound tasks at a time, but even 5 simultaneous CPU-bound tasks cause CPU throttling. So it is useful to have different workers with own limits for CPU-bound and I/O-bound tasks. The simplest way to reach this goal is to make ability to pass max_jobs as CLI-argument for worker.

Separating cron scheduler from worker solves several problems:

1. In case where we have many workers each worker trying to add the same cron jobs, which may cause duplicate task run
2. Overhead for run_cron job on every worker
3. It may be difficult to implement state storing for scheduler (like https://github.com/sibson/redbeat for Celery)

I have a problem from time to time when queue is stuck on some message without any progress until most of the messages are invalidated by time. Restart or adding/removing clients manually don't help. Strange thing is CPU/RAM usage stays the same.
Maybe there is some kind of limitation when too many records in queue?
If anyone could help me at least how to debug a problem, because messages are not processing and logger has no new records either.
Only such records from time to time

redis_version=6.2.4 mem_usage=3.23G clients_connected=536 db_keys=6685280
recording health: Mar-31 10:04:29 j_complete=755 j_failed=1 j_retried=0 j_ongoing=0 queued=511

P.S. thanks for a great project. Have not seen anything better for asyncio

It would be great if there was an opportunity to pass arguments to task during creating cron job

Example:

@darq.task
async def print_name(name: str = 'John'):
    print(name)


cron_jobs = (
    cron(print_name, hour=3, task_kwargs=dict(name='Lucas')),
)

Fix test_max_bursts_multiple, because it fails sometimes

Are there any plans to support rate limiting?

Currently, if darq don't found function for job it logs warning, while error would be more appropriate in that case, I think.