pjongy / jasyncq

High reliability asynchronous task queue using mysql

jasyncq's Issues

Task sequence (order)

Sometime there should be gone in order of each tasks.
For example, B task should be done after A regardless any task is between them like :
A ..K(independent).. B(depends on A)

Prob.
jasyncq does not support their ordering and but multiple consumers are supported.
In this case, below situation could be occurred.

1. Publish task A
2. Publish task K (indenpendent)
3. Publish task B (depends on A)
4. Worker 1 consumes ONE task (A fetched)
5. Worker 2 consumes TWO task (K, B fetched)
6. Worker 1 shutdowned (A task pended)
7. Worker do their job (K- done, B - done (without A) )

Sol.
It can be solved by save tasks with their pre-done task number like:

| task_id | queued_at | depends_on |
|    A    |  2021-01  |            |
|    K    |  2021-02  |            |
|    B    |  2021-03  |     A      |

and fetch query can be:

# Does not fetch if depends_on id remains in task queue
SELECT child.*
  FROM queue_table as child
LEFT JOIN queue_table as parent
  ON child.depends_on=parent.uuid
WHERE parent.uuid IS NULL;

Result rows' will be returned if dependent task is already done or independent task.

Executing queries is slow

Executing just one query by joining all query list sql would be better than iterating for multiple queries. (It leass low performance.)

For example:

    async def _execute(self, queries: List[str]):
        async with self.pool.acquire() as conn:
            conn: Connection = conn  # NOTE(pjongy): For type hinting
            async with conn.cursor() as cur:
                cur: Cursor = cur  # NOTE(pjongy): For type hinting
                await cur.execute(';'.join(queries))
                await conn.commit()

is better than it

jasyncq/jasyncq/repository/abstract.py

Lines 14 to 19 in c2d681b

	cur: Cursor = cur # NOTE(pjongy): For type hinting
	[
	await cur.execute(clause)
	for clause in queries
	]
	await conn.commit()

Implement urgent priority

Separate example code

as-is: exmple/main.py runs publish and pull
to-be separate pub.py and worker.py