litements / litequeue Goto Github PK

Is it possible to handle more Queues in on db by adding a Tablename?

BR, NiSche

https://litements.exampl.io/queue/

It seems the actual link should be: https://litements.polyrand.net/queue/

Hello,

Annotation is str, but bytes works, maybe change annotation to str|bytes?

In [1]: from litequeue import LiteQueue

In [2]: q = LiteQueue(memory=True)

In [3]: q.put('a')
Out[3]: Message(data='a', message_id='0667127d-c488-75f4-8000-9b1e8b8dbb25', status=<MessageStatus.READY: 0>, in_time=1718691804283328000, lock_time=None, done_time=None)

In [4]: q.put(b'a')
Out[4]: Message(data=b'a', message_id='0667127e-00ab-7d4c-8000-47674cc35d04', status=<MessageStatus.READY: 0>, in_time=1718691808041979000, lock_time=None, done_time=None)

In [5]: q.pop()
Out[5]: Message(data='a', message_id='0667127d-c488-75f4-8000-9b1e8b8dbb25', status=1, in_time=1718691804283328000, lock_time=1718691813868331000, done_time=None)

In [6]: q.pop()
Out[6]: Message(data=b'a', message_id='0667127e-00ab-7d4c-8000-47674cc35d04', status=1, in_time=1718691808041979000, lock_time=1718691814732661000, done_time=None)

In [7]: q.pop()

When calling done(), lastrowid always evaluates to 0, which causes "assert x" to raise an AssertionError exception.

This is happening using Python 3.10.12 and sqlite3 3.42.0

Firstly, thank you very much for making this open source project.

I was wondering if there is any retry mechanism for items that haven't been marked done explicitly?

If you pop item, then this happen:
UPDATE Queue SET status = 1, lock_time = strftime('%s','now') WHERE message_id = ? and status =0

When you mark item as done then this happen:
UPDATE Queue SET status = 2, done_time = strftime('%s','now') WHERE message_id = ?

... but if the queue worker fails and doesn't mark item done, what happens next? Would the item with status=1 stay forever in the queue?

When trying to pop a message from a queue the following exception is given

_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress

Messages can be added without error.

I read the source code and I think using EXCLUSIVE transactions, the answer is affirmative, but can I have one Python process running that adds to the queue then multiple others reading and working? It would have to rely on sqlite's locking but I think the transaction does that.

I am looking on the _pop_transaction and wondering how it works and what guarantees it has.

In order for it to pop the item it executes this:

BEGIN IMMEDIATE

SELECT message, message_id FROM Queue
            WHERE rowid = (SELECT min(rowid) FROM Queue
                           WHERE status = 0)

UPDATE Queue SET status = 1, lock_time = strftime('%s','now') WHERE message_id = :message_id AND status = 0

Assuming there is more than one worker that pops the items from the queue, is it possible for them to SELECT the same message_id? If that happens then they would both try to UPDATE the status to 1.
Only one of them would actually do the actual row update, since there is WHERE status = 0 check, so the first UPDATE wins... but in fact shouldn't the code check the result: select changes(); to discard the item which actually hasn't effectively updated DB?

What's the role of IMMEDIATE locking here by the way? I've just started looking on the SQLite project and come from the MySQL background, pardon my lack of experience.

im new to these message queue and after skimming the docs as far as i know, i need to implement this by priority therefore a priority queue not a normal queue. or maybe im missing something?

Is it possible for this thing to implement a ratelimit mechanism?
like node js p-queue