litements / litequeue Goto Github PK
View Code? Open in Web Editor NEWQueue built on top of SQLite
Home Page: https://litements.polyrand.net/queue/
License: MIT License
Queue built on top of SQLite
Home Page: https://litements.polyrand.net/queue/
License: MIT License
Is it possible to handle more Queues in on db by adding a Tablename?
BR, NiSche
https://litements.exampl.io/queue/
It seems the actual link should be: https://litements.polyrand.net/queue/
Hello,
Line 278 in 90e9694
Annotation is str, but bytes works, maybe change annotation to str|bytes?
In [1]: from litequeue import LiteQueue
In [2]: q = LiteQueue(memory=True)
In [3]: q.put('a')
Out[3]: Message(data='a', message_id='0667127d-c488-75f4-8000-9b1e8b8dbb25', status=<MessageStatus.READY: 0>, in_time=1718691804283328000, lock_time=None, done_time=None)
In [4]: q.put(b'a')
Out[4]: Message(data=b'a', message_id='0667127e-00ab-7d4c-8000-47674cc35d04', status=<MessageStatus.READY: 0>, in_time=1718691808041979000, lock_time=None, done_time=None)
In [5]: q.pop()
Out[5]: Message(data='a', message_id='0667127d-c488-75f4-8000-9b1e8b8dbb25', status=1, in_time=1718691804283328000, lock_time=1718691813868331000, done_time=None)
In [6]: q.pop()
Out[6]: Message(data=b'a', message_id='0667127e-00ab-7d4c-8000-47674cc35d04', status=1, in_time=1718691808041979000, lock_time=1718691814732661000, done_time=None)
In [7]: q.pop()
When calling done(), lastrowid always evaluates to 0, which causes "assert x" to raise an AssertionError exception.
This is happening using Python 3.10.12 and sqlite3 3.42.0
Line 379 in 156177d
Firstly, thank you very much for making this open source project.
I was wondering if there is any retry mechanism for items that haven't been marked done
explicitly?
If you pop item, then this happen:
UPDATE Queue SET status = 1, lock_time = strftime('%s','now') WHERE message_id = ? and status =0
When you mark item as done
then this happen:
UPDATE Queue SET status = 2, done_time = strftime('%s','now') WHERE message_id = ?
... but if the queue worker fails and doesn't mark item done
, what happens next? Would the item with status=1
stay forever in the queue?
When trying to pop a message from a queue the following exception is given
_sqlite3.OperationalError: cannot commit transaction - SQL statements in progress
Messages can be added without error.
I read the source code and I think using EXCLUSIVE
transactions, the answer is affirmative, but can I have one Python process running that adds to the queue then multiple others reading and working? It would have to rely on sqlite's locking but I think the transaction does that.
I am looking on the _pop_transaction
and wondering how it works and what guarantees it has.
In order for it to pop the item it executes this:
BEGIN IMMEDIATE
SELECT message, message_id FROM Queue
WHERE rowid = (SELECT min(rowid) FROM Queue
WHERE status = 0)
UPDATE Queue SET status = 1, lock_time = strftime('%s','now') WHERE message_id = :message_id AND status = 0
Assuming there is more than one worker that pops
the items from the queue, is it possible for them to SELECT
the same message_id
? If that happens then they would both try to UPDATE
the status
to 1
.
Only one of them would actually do the actual row update, since there is WHERE status = 0
check, so the first UPDATE wins... but in fact shouldn't the code check the result: select changes();
to discard the item which actually hasn't effectively updated DB?
What's the role of IMMEDIATE
locking here by the way? I've just started looking on the SQLite project and come from the MySQL background, pardon my lack of experience.
im new to these message queue and after skimming the docs as far as i know, i need to implement this by priority therefore a priority queue not a normal queue. or maybe im missing something?
Is it possible for this thing to implement a ratelimit mechanism?
like node js p-queue
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.