omnilib / aiosqlite

asyncio bridge to the standard sqlite3 module

aiosqlite: Sqlite for AsyncIO

aiosqlite provides a friendly, async interface to sqlite databases.

It replicates the standard sqlite3 module, but with async versions of all the standard connection and cursor methods, plus context managers for automatically closing connections and cursors:

async with aiosqlite.connect(...) as db:
    await db.execute("INSERT INTO some_table ...")
    await db.commit()

    async with db.execute("SELECT * FROM some_table") as cursor:
        async for row in cursor:
            ...

It can also be used in the traditional, procedural manner:

db = await aiosqlite.connect(...)
cursor = await db.execute('SELECT * FROM some_table')
row = await cursor.fetchone()
rows = await cursor.fetchall()
await cursor.close()
await db.close()

aiosqlite also replicates most of the advanced features of sqlite3:

async with aiosqlite.connect(...) as db:
    db.row_factory = aiosqlite.Row
    async with db.execute('SELECT * FROM some_table') as cursor:
        async for row in cursor:
            value = row['column']

    await db.execute('INSERT INTO foo some_table')
    assert db.total_changes > 0

Install

aiosqlite is compatible with Python 3.8 and newer. You can install it from PyPI:

$ pip install aiosqlite

Details

aiosqlite allows interaction with SQLite databases on the main AsyncIO event loop without blocking execution of other coroutines while waiting for queries or data fetches. It does this by using a single, shared thread per connection. This thread executes all actions within a shared request queue to prevent overlapping actions.

Connection objects are proxies to the real connections, contain the shared execution thread, and provide context managers to handle automatically closing connections. Cursors are similarly proxies to the real cursors, and provide async iterators to query results.

License

aiosqlite is copyright Amethyst Reese, and licensed under the MIT license. I am providing code in this repository to you under an open source license. This is my personal repository; the license you receive to my code is from me and not from my employer. See the LICENSE file for details.

aiosqlite's Issues

How to integration with aiohttp.web?

Hi,
Dear developer. I has a question.
How to integration with aiohttp.web?
I want to use it at app on_startup.

thanks.

cursor.lastrowid sometimes returning zero instead of the correct value

Description

I'm seeing somewhat unreliable behaviour with the lastrowid property of a cursor. I'm doing INSERT ... ON CONFLICT DO UPDATE statements on multiple rows:

    for rel in releases:
        for file in releases[rel]:
            async with db.execute(PROJECT_FILES_SQL, project_files_args(file)) as cursor:
                file_id = cursor.lastrowid
                print(file_id, name, file.get("filename"))

What I'm seeing is many rows with a file_id of zero, but when I query the database directly, the file_id is not zero, and has a perfectly valid value.

Unfortunately, I cannot reproduce this in a smaller test case. I suspect that this could be a race condition, with the lastrowid value being "lost" because another statement is executed on the database thread before lastrowid is called, but I don't have enough understanding of asyncio to be sure if that's possible.

Details

• OS: Windows 10
• Python version: 3.8.5
• aiosqlite version: 0.15.0
• Can you repro on 'main' branch? Not tried
• Can you repro in a clean virtualenv? Not tried

Program stuck on __aenter__ failure

MacOsX Mojave, Python 3.7, aiosqlite 0.10

If Connection.__aenter__() fails for any reason, the thread is not shut down properly, so the Python interpreter will print the stack trace and get stuck forever instead of exiting 1.

Demonstration:

import asyncio
import aiosqlite

async def main():
    async with aiosqlite.connect("/permission_error.db"):
        pass


asyncio.run(main())

fetchall() doesn't return anything

Description

I was working on a discord bot and needed to use aiosqlite for storing configs. however when i check the sqlite database with:
async def is_guild_owner(ctx): async with aiosqlite.connect(vars.database) as conn: allowed_roles = await conn.execute(f'''SELECT role_id FROM 'Allowed_roles' WHERE server_id={int(ctx.guild.id)};''') if(ctx.message.author == ctx.guild.owner or ctx.message.author.id in await allowed_roles.fetchall()): await conn.close() return True else: for role in ctx.message.author.roles: if role.id in await allowed_roles.fetchall(): return True print("|None of theses: "+ str(ctx.message.author.roles) + " are in: "+ str(await allowed_roles.fetchall())) await allowed_roles.close() return False
It returns false everytime and when i print out the allowed_roles.fetchall() it shows an empty list and i know i have a record inside of the database with the proper table.

Details

• OS: Windows 10
• Python version: Python 3.6.6
• aiosqlite version: 0.7.0
• Can you repro on master? umm no?
• Can you repro in a clean virtualenv? idk how to change environments

Exiting with ctrl-c

Description

For example, if a script

conn = await aiosqlite.connect('db.sqlite')
cursor = await conn.cursor()
... some work with cursor ...

is interrupted with ctrl-c and it does not exit to the console. If conn and cursor are closed before ctrl-c or inside try-except catching ctrl-c, it does.

However, it can be tricky with multiple cursors and db connections in different parts of a program, to know and close them properly when interrupted with ctrl-c.

What would be the best way to handle KeyboardInterrupt and aiosqlite? Or, would it be possible that aiosqlite does not hang when interrupted before closing connections? aiosqlite3 does not show this behavior.

Details

• OS: Windows as well as WSL
• Python version: 3.7
• aiosqlite version: 0.15.0
• Can you repro on master? I don't know how to install aiosqlite with master since I don't see setup.py there.
• Can you repro in a clean virtualenv? Yes

Long lived connection

Hi,

This isn't as much of a bug, so doing away with the template, apologies for that

I was trying to use aiosqlite with a simple web server app and would have liked to use something like a app wide connection or a pool of connections. However couldn't find much on how to get this done. Digging into the code saw context manager was used to control the thread.

Couple of questions to help me understand and use the lib better

1. Is it unsafe for multiple coroutines (async defs) to be using a single sqlite DB connection thread? Anything found while developing leading to make the design decision?
2. If no, would it be possible to have code in the Connection class to provide long living connection? (If yes I'd be willing to work on a PR)

If I missed something which could be used for the above use case, please help me understand it better

ValueError on db.execute('select...') with parameters

Description

Raise ValueError exception on execute select statement with parameters.

Details

Linux 4.4.192-rockchip64/Python 3.6.9/aiosqlite 0.11.0 or
Windows 7 x64/Python 3.8.0/aiosqlite 0.11.00

import asyncio
import aiosqlite


async def task1(num):
    async with aiosqlite.connect('test.sqlite') as db:
        await db.execute('''
            create table test (id integer)
            ''')
        await db.execute('''
            insert into test (id) values(?)
            ''', (num))
        cur = await db.execute('''
            select id from test where id = ?
            ''', (num)) # ValueError !!!!!!!!

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(task1(10)))

Support a set_trace_callback proxy

Description

Support for set_trace_callback is missing.

Details

• OS: All
• Python version: All
• aiosqlite version: All
• Can you repro on master? N/A
• Can you repro in a clean virtualenv? N/A

Threading model

I'm a little confused by the wording in the readme. Does this It does this by using a single, shared thread per connection. mean there is a single thread shared between all connections or there is one thread per connection (but how is it then shared ?). How many threads are started ?

Is this library in any way related to Write-Ahead Logging?

Dependency on typing_extensions is completely unnecessary

AsyncContextManager has been already present since 3.5.4 and aiosqlite requires Python >= 3.6.
This patch removes it.

(and AsyncContextManager has been deprecated since 3.9, so you should remove it anyway).

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

__aiter__() and fetchone() are 15x slower than sqlite3

• OS: MacOS Mojave
• Python version: Anaconda Python 3.7
• aiosqlite version: 0.10.0 (conda-forge)

In sqlite3, __iter__(), fetchone(), fetchmany() and fetchall() offer identical performance.
In aiosqlite, fetchmany() and fetchall() offer virtually the same performance as in sqlite3, but __aiter__() and fetchone() are 15 times slower due to the fact that they send individual calls to sqlite3 fetchone() through thread synchronization.

They should be reimplemented as wrappers around fetchmany() with a small-ish hardcoded chunk size. A 250 rows buffer is a very small price to pay memory-wise, and makes performance indistinguishable from sqlite3.
All fetch methods (fetchone(), fetchmany(), fetchall(), and __aiter__()) would have to be tweaked to hide the wrapper-level caching in the unlikely event that somebody mixes them within the same cursor.

Benchmarks (reading a table with 120k rows from SSD):

import asyncio
import sqlite3
import time
import aiosqlite

DBNAME = 'mydatabase.db'
QUERY = 'SELECT * FROM mytable'
CHUNKSIZE = 250


def bench(label, func, db):
    t1 = time.time()
    func(db)
    t2 = time.time()
    print(label, t2 - t1)


async def abench(label, func, db):
    t1 = time.time()
    await func(db)
    t2 = time.time()
    print(label, t2 - t1)


def sqlite_iter(db):
    cursor = db.execute(QUERY)
    for row in cursor:
        pass
    cursor.close()


def sqlite_fetchone(db):
    cursor = db.execute(QUERY)
    while True:
        row = cursor.fetchone()
        if not row:
            break
    cursor.close()


def sqlite_fetchmany(db):
    cursor = db.execute(QUERY)
    while True:
        rows = cursor.fetchmany(CHUNKSIZE)
        if not rows:
            break
        for row in rows:
            pass
    cursor.close()


async def aiosqlite_aiter(db):
    async with db.execute(QUERY) as cursor:
        async for row in cursor:
            pass


async def aiosqlite_fetchone(db):
    async with db.execute(QUERY) as cursor:
        while True:
            row = await cursor.fetchone()
            if not row:
                break


async def aiosqlite_fetchmany(db):
    async with db.execute(QUERY) as cursor:
        while True:
            rows = await cursor.fetchmany(CHUNKSIZE)
            if not rows:
                break
            for row in rows:
                pass


async def amain():
    with sqlite3.connect(DBNAME) as db:
        bench('sqlite3 __iter__', sqlite_iter, db)
        bench('sqlite3 fetchone', sqlite_fetchone, db)
        bench('sqlite3 fetchmany', sqlite_fetchmany, db)

    async with aiosqlite.connect(DBNAME) as db:
        await abench('aiosqlite __aiter__', aiosqlite_aiter, db)
        await abench('aiosqlite fetchone', aiosqlite_fetchone, db)
        await abench('aiosqlite fetchmany', aiosqlite_fetchmany, db)


if __name__ == '__main__':
    asyncio.run(amain())

Output:

sqlite3 __iter__ 0.4601600170135498
sqlite3 fetchone 0.46087217330932617
sqlite3 fetchmany 0.4549098014831543
aiosqlite __aiter__ 7.60332989692688
aiosqlite fetchone 7.877390146255493
aiosqlite fetchmany 0.49387431144714355

Exception handling

Description

Database is locked when there is an exception.

PS: I am using databases where I don't get access to underlying connection and cursor.

Details

Minimal reproducible example:

import aiosqlite

async def test():
    db = await aiosqlite.connect("data.db")

    create_sql = """
    CREATE TABLE edits (
    id INTEGER NOT NULL,
    category TEXT NOT NULL,
    question TEXT NOT NULL,
    answer TEXT NOT NULL,
    action VARCHAR(6) NOT NULL,
    CONSTRAINT pk_edits PRIMARY KEY (id),
    CONSTRAINT ck_edits_action CHECK (action  in ('INSERT', 'UPDATE', 'DELETE')))
    """

    data = {
        "category": "cat1",
        "question": "hello",
        "answer": "world",
        "action": "USDATE",
    }

    columns = ", ".join(data.keys())
    sql = "INSERT INTO edits ({}) VALUES {}".format(columns, tuple(data.values()))
    cursor = await db.execute(create_sql)
    cursor = await db.execute(sql)
    await cursor.close()
    await db.commit()
    await db.close()

start ipython and execute await test().

I am correctly getting the IntegrityError: CHECK constraint failed: ck_edits_action. But then I cannot exit ipython and accessing from another process raises db locked error.

• OS: Windows 7
• Python version: 3.6.8
• aiosqlite version: 0.11.0
• Can you repro on master? NA
• Can you repro in a clean virtualenv? Yes

Deterministic function support

In Python 3.8, sqlite3 added support for creating deterministic SQL functions via Connection.create_function. Adding this to aiosqlite can be achieved with a simple patch to pass through the deterministic kwarg:

diff --git a/aiosqlite/core.py b/aiosqlite/core.py
index 0135654..4992d79 100644
--- a/aiosqlite/core.py
+++ b/aiosqlite/core.py
@@ -196,12 +196,20 @@ class Connection(Thread):
         """Interrupt pending queries."""
         return self._conn.interrupt()
 
-    async def create_function(self, name: str, num_params: int, func: Callable) -> None:
+    async def create_function(
+        self, name: str, num_params: int, func: Callable, deterministic: bool = False,
+    ) -> None:
         """Create user-defined function that can be later used
         within SQL statements. Must be run within the same thread
         that query executions take place so instead of executing directly
         against the connection, we defer this to `run` function."""
-        await self._execute(self._conn.create_function, name, num_params, func)
+        await self._execute(
+            self._conn.create_function,
+            name,
+            num_params,
+            func,
+            deterministic=deterministic,
+        )
 
     @property
     def in_transaction(self) -> bool:
diff --git a/aiosqlite/tests/smoke.py b/aiosqlite/tests/smoke.py
index af6c239..243cff2 100644
--- a/aiosqlite/tests/smoke.py
+++ b/aiosqlite/tests/smoke.py
@@ -313,6 +313,21 @@ class SmokeTest(aiounittest.AsyncTestCase):
                 row = await res.fetchone()
                 self.assertEqual(row[0], 20)
 
+    async def test_create_function_deterministic(self):
+        """Assert that after creating a deterministic custom function, it can be used.
+
+        https://sqlite.org/deterministic.html
+        """
+
+        def one_arg(num):
+            return num * 2
+
+        async with aiosqlite.connect(TEST_DB) as db:
+            await db.create_function("one_arg", 1, one_arg, deterministic=True)
+            await db.execute("create table foo (id int, bar int)")
+            # Non-deterministic functions cannot be used in indexes
+            await db.execute("create index t on foo(one_arg(bar))")
+
     async def test_set_trace_callback(self):
         statements = []

The above patch passes the test suite on Python 3.8.

However, the deterministic kwarg is only available on Python >= 3.8 and aiosqlite supports >= 3.6. Is it possible to include support for this on only Python 3.8 installs? Perhaps conditionally defining aiosqlite.Connection.create_function with a different signature based on sys.version_info?

Don't log result in DEBUG, or truncate if too large

Description

Logging the result of the run method can be problematic when the result is big.
If you query a whole table, the log line can quickly be really, really heavy!! Like in the 100MB range 😮

aiosqlite/aiosqlite/core.py

Line 99 in b1a7ea3

LOG.debug("returning %s", result)

We are currently facing downtime on a webservice using this library indirectly and setting the log level to DEBUG.

I suggest to stop logging the result in DEBUG level (or any other level), or at least truncate it when it's too big.
"Too big" needs to be defined though. Or maybe always truncate to... 100 or 1K characters?

Details

• OS: RedHat
• Python version: 3.6.12
• aiosqlite version: 0.16.0
• Can you repro on 'main' branch? line is still in HEAD, so most probably
• Can you repro in a clean virtualenv? not relevant?

Please release v0.3.1

Hi @jreese
The current master has 2 desirable features over the latest pypi release:

1. passing in a custom event loop
2. better performance

Could you please release v0.3.1 for us?

Thanks in advance

Test coverage is < 100%

Description

Coverage is below 100%. We should fix that.

Performance enhancements

I ran into similar problems as found in 48e3021 regarding performance issues. I noticed it especially with processes that ran many, small queries.

I wasn't particularly happy with the _execute() method blocking the event loop at all, and just polling by adding asyncio.sleep calls means spending a heap of time sleeping.

I rewrote the run() and _execute(). The result is a simpler thread loop, commands to be executed can be queued (I removed the asyncio.Lock), and awaiting a result no longer relies on polling, so results become available almost instantly after being processed in the SQLite thread.

My code:

class FasterConnection(aiosqlite.Connection):
    def run(self):
        while self._running:
            try:
                future, function = self._tx.get(timeout=0.1)
            except queue.Empty:
                continue

            try:
                self._loop.call_soon_threadsafe(future.set_result, function())
            except BaseException as e:
                self._loop.call_soon_threadsafe(future.set_exception(e))

    async def _execute(self, fn, *args, **kwargs):
        function = functools.partial(fn, *args, **kwargs)
        future = self._loop.create_future()

        self._tx.put_nowait((future, function))

        return await future

I know this isn't the traditional way of submitting code to a project, I apologise. Hopefully it's useful, however.

Support for DB-API 2.0 style cursor-driven control

Description

Per https://docs.python.org/3/library/sqlite3.html this style allows cursor vs. connection to drive execution control. This is convenient when chaining multiple actions together naked or under (nested) savepoints.

Details

• OS: All
• Python version: All
• aiosqlite version: All
• Can you repro on master? N/A
• Can you repro in a clean virtualenv? N/A

[withdrawn]

[withdrawn]

Transactions are difficult on shared connections

Description

Currently, the library presumes that any sequential set of operations should be allowed to happen concurrently with any other set of operations. This is incompatible with transactional operations.

Here's a simple program to demonstrate the existing behavior and the desired behavior:

import aiosqlite                                                                                                 
import asyncio                                                                                                   
import os                                                                                                        
                                                                                                                 
async def insert_100_error(conn,val):                                                                            
    async with conn.cursor() as c:                                                                               
        await c.execute('BEGIN')                                                                                 
        await c.executemany("INSERT INTO foo VALUES (?)", [(val,) for x in range(100)])                          
        await c.execute("INSERT INTO foo VALUES (?)", (val + 'done',))                                           
        await c.execute('COMMIT')                                                                                
                                                                                                                 
async def insert_100_helper(conn,val):                                                                           
    def my_operation(c):                                                                                         
        c.execute('BEGIN')                                                                                       
        c.executemany("INSERT INTO foo VALUES (?)", [(val,) for x in range(100)])                                
        c.execute("INSERT INTO foo VALUES (?)", (val + 'done',))                                                 
        c.execute('COMMIT')                                                                                      
    async with conn.cursor() as c:                                                                               
        await c._execute(my_operation,c._cursor)                                                                 
                                                                                                                 
async def run():                                                                                                 
    async with aiosqlite.connect('test_linear.db') as conn:                                                      
        await conn.execute("CREATE TABLE foo (val1 text)")                                                       
        await conn.commit()                                                                                      
        await asyncio.gather(*[insert_100_error(conn,x) for x in ['a','b','c','d']])                             
        #await asyncio.gather(*[insert_100_helper(conn,x) for x in ['a','b','c','d']])                           
                                                                                                                 
                                                                                                                 
os.remove('test_linear.db')                                                                                      
asyncio.get_event_loop().run_until_complete(run())                                                                                                           

So rather than submit each individual function on a cursor to the sqlite thread, we submit a callback that takes a plain cursor which can be used directly.

I don't know if there are significant drawbacks to this approach, but it works the way I expect it to whereas I can't see a way to accomplish this with the library right now.

Not sure what you'd call such a function.

Cursor.executelinear? executecb? executesync?

If I've just missed the right way to do this with the current API, please let me know.

Details

• OS: Linux
• Python version: 3.6
• aiosqlite version: v0.8
• Can you repro on master? Yes
• Can you repro in a clean virtualenv? Yes

[solved] Parametrized UPDATE doesn't work

Description

I have trouble making UPDATE request to database when I try to pass parameters to SQL query

Database scheme:

CREATE TABLE "versions" (
	"id"	INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE,
	"build_type"	TEXT NOT NULL,
	"build_version"	TEXT NOT NULL DEFAULT 'Unknown'
);
INSERT INTO "versions" VALUES (1,'beta','Unknown'), (2,'stable','Unknown');

This code works fine:

async def update_version(build_type: str, new_version: str):
    statement = "UPDATE versions SET build_version='aaa' WHERE build_type='beta'"
    async with aiosqlite.connect("database.db") as db:
        a = await db.execute(statement)
        print(a.rowcount)  # equals 1
        await db.commit()

However this code doesn't change anything:

async def update_version(build_type: str, new_version: str):
    statement = "UPDATE versions SET build_version=? WHERE build_type=?"
    async with aiosqlite.connect("database.db") as db:
        a = await db.execute(statement, [build_type, new_version])
        print(a.rowcount)  # equals 0
        await db.commit()

How I run this code:

if __name__ == '__main__':
    import asyncio
    asyncio.run(update_version("beta", "bbb"))

Details

• OS: Manjaro KDE
• Python version: 3.8.5
• aiosqlite version: 0.15.0
• Can you repro on master? Hm...yes?
• Can you repro in a clean virtualenv? Yes

pass parameter (bytes) to aiosqlite.connect()

Description

async with aiosqlite.connect(b'name.db') as db:

is different from

async with aiosqlite.connect('name.db') as db:

It seems that dealing with bytes has some problem. Thank you!

Details

• OS: linux (Manjaro)
• Python version: 3.7.1
• aiosqlite version: v0.8
• Can you repro on master? do not know
• Can you repro in a clean virtualenv? do not know

Stop logging "returning exception"

Every time an exception occurs, the following line is logged if logging is enabled:

aiosqlite/aiosqlite/core.py

Lines 98 to 99 in 55558d4

	except BaseException as e:
	LOG.exception("returning exception %s", e)

This is bad because the library can and does propagate the error:

aiosqlite/aiosqlite/core.py

Line 100 in 55558d4

get_loop(future).call_soon_threadsafe(future.set_exception, e)

The user code will handle it, there is no need to log it unless the library could not pass it to the user's code. If the user code wants to log it, that's their job. Maybe the INFO level would be more appropriated "hey, an error occured and I am informing that I will hand it over to user's code". Not ERROR, it works fine!

misleading return type for cursor.fetchall()

cursor.fetchall() marked as "return Iterable[sqlite3.Row]" what is misleading analyzer

may be better List[sqlite3.Row], because list is always returned there

the same for fetchmany

Tests failing on PyPy

Description

Test suite failing on PyPy:

(venv_pypy3) user@dt1:~/aiosqlite$ python3 --version
Python 3.5.3 (eb6e70a9658e, Apr 19 2019, 14:30:38)
[PyPy 7.1.0-alpha0 with GCC 6.3.0 20170516]
(venv_pypy3) user@dt1:~/aiosqlite$ python3 -m coverage run -m aiosqlite.tests
test_connection_await (aiosqlite.tests.smoke.SmokeTest) ... ok
test_connection_context (aiosqlite.tests.smoke.SmokeTest) ... ok
test_connection_locations (aiosqlite.tests.smoke.SmokeTest) ... ok
test_connection_properties (aiosqlite.tests.smoke.SmokeTest) ... FAIL
test_context_cursor (aiosqlite.tests.smoke.SmokeTest) ... ok
test_enable_load_extension (aiosqlite.tests.smoke.SmokeTest)
Assert that after enabling extension loading, they can be loaded ... skipped "python was not compiled with sqlite3 extension support, so we can't test it"
test_fetch_all (aiosqlite.tests.smoke.SmokeTest) ... ok
test_iterable_cursor (aiosqlite.tests.smoke.SmokeTest) ... ok
test_multiple_connections (aiosqlite.tests.smoke.SmokeTest) ... ok
test_multiple_queries (aiosqlite.tests.smoke.SmokeTest) ... ok

======================================================================
FAIL: test_connection_properties (aiosqlite.tests.smoke.SmokeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/venv_pypy3/site-packages/aiounittest/helpers.py", line 139, in wrapper
    return loop.run_until_complete(future)
  File "/opt/pypy3-deb9-yarmak/lib-python/3/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "/opt/pypy3-deb9-yarmak/lib-python/3/asyncio/futures.py", line 293, in result
    raise self._exception
  File "/opt/pypy3-deb9-yarmak/lib-python/3/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/user/aiosqlite/aiosqlite/tests/smoke.py", line 169, in test_connection_properties
    self.assertEqual(db.text_factory, str)
AssertionError: <function _unicode_text_factory at 0x00005603b32ac7a0> != <class 'str'>

----------------------------------------------------------------------
Ran 10 tests in 1.148s

FAILED (failures=1, skipped=1)

Not to mention typed-ast doesn't builds on PyPy, but it's not mandatory for test suite.

Details

• OS: Debian 9
• Python version: Python 3.5.3 (eb6e70a9658e, Apr 19 2019, 14:30:38) [PyPy 7.1.0-alpha0 with GCC 6.3.0 20170516] on linux
• aiosqlite version: 0.10.0
• Can you repro on master?
• Can you repro in a clean virtualenv?

InvalidStateError raised

Description

TL;DR I think there is a race condition in setting the future's result using get_loop(future).call_soon_threadsafe(future.set_result, result) causing the InvalidStateError to be raised if the future is already cancelled/timeout

I am running sqlite on my embedded PC for pulling messages from the SQLite database

    async def get_all_unsent(self, query_batch: int = 100) -> typing.AsyncGenerator:
        _id = 0
        while True:
            results_q = await self.conn.execute(
                "SELECT * from mqtt WHERE mqtt.id > ? ORDER BY id LIMIT ?",
                (_id, query_batch),
            )
            headings = [desc[0] for desc in results_q.description]
            results = await results_q.fetchall()
            if not results:
                break
            for res in results:
                res = {key: value for key, value in zip(headings, res)}
                _id = res.pop("id")
                yield res

In my logs I got the following errors

handle: <Handle Future.set_result(<sqlite3.Curs...at 0xa0e887a0>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.base_futures.InvalidStateError: invalid state
2020-09-02 23:37:41,690 MainThread      ERROR    base_events    :1604     Exception in callback Future.set_result(<sqlite3.Curs...at 0xa0da0b60>)
handle: <Handle Future.set_result(<sqlite3.Curs...at 0xa0da0b60>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.base_futures.InvalidStateError: invalid state
2020-09-02 23:44:08,571 MainThread      ERROR    base_events    :1604     Exception in callback Future.set_result(<sqlite3.Curs...at 0xa0da9460>)
handle: <Handle Future.set_result(<sqlite3.Curs...at 0xa0da9460>)>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
asyncio.base_futures.InvalidStateError: invalid state

Looking through the code I noticed that there is a possible race condition due to the main run loop running in a thread. If the future is cancelled before the threadsafe call_soon sets it, we would get an InvalidStateError.

def run(self) -> None:
        """
        Execute function calls on a separate thread.
        :meta private:
        """
        while self._running:
            try:
                future, function = self._tx.get(timeout=0.1)
            except Empty:
                continue

            try:
                LOG.debug("executing %s", function)
                result = function()
                LOG.debug("returning %s", result)
                get_loop(future).call_soon_threadsafe(future.set_result, result)
            except BaseException as e:
                LOG.exception("returning exception %s", e)
                get_loop(future).call_soon_threadsafe(future.set_exception, e)

One unanswered question I have is that I didn't do anything to explicitly cancel or set the future's result and so I'm not sure how it got into this InvalidState.

Details

• OS: Debian 8
• Python version: 3.7.4
• aiosqlite version: 0.15.0
• Can you repro on 'main' branch? Hard to reproduce
• Can you repro in a clean virtualenv? Hard to reproduce

how do I call create_function?

This fails with a cross-thread violation:

db._conn.create_function, "NEEDS_UPDATE", 5,
                 lambda id, name, original, update, activation:
                 queue.put_nowait({
                     "claimID": id, "name": name, "originalHeight": original,
                     "updateHeight": update, "activationHeight": activation
                 })

If I then put that into await db._execute(...) it fails with this error:

  File "/home/brannon/Workspace/lbry-sdk/venv/lib/python3.7/site-packages/aiosqlite/core.py", line 167, in _execute
    return await future
  File "/home/brannon/Workspace/lbry-sdk/venv/lib/python3.7/site-packages/aiosqlite/core.py", line 153, in run
    result = function()
TypeError: argument 1 must be str, not list

Recommended way load sqlite extensions?

Description

I'm trying to use spatialite, and need to load an extension to do so. In sqlite3, this is done with the enable_load_extension function, however it seems like that isn't exposed in aiosqlite. I'm able to implement this myself if it isn't currently possible, but would appreciate some pointers.

Thanks! :)

Details

• OS: MacOS
• Python version: 3.6.7
• aiosqlite version: latest

Cant try except sqlite3.IntegrityError

If I try to create a row with a always existing primary key, I get following error:

sqlite3.IntegrityError: UNIQUE constraint failed

This is absolutly correct. But even with imported error module (from aiosqlite import Error), I cant supress this error message with a
try: create_row except: pass

Details

• OS: Windows 10 2004 / Ubuntu 19.10
• Python version: 3.7.7
• aiosqlite version: 0.13.0
• Can you repro on master?
• Can you repro in a clean virtualenv? yes

Expose connection outside context manager

Hello,

I'm using aiosqlite and it works well for me, but I can suggest useful improvement.

Currently aiosqlite creates thread and connection when user enters created context manager and destroys thread and connection when user leaves context. It means user have to reopen database and run all connection setup pragma's every time I access database. In order to reuse connection and thread I have to invoke __aenter__ and __aexit__ directly.

Why connection reuse is important:

• Database open and thread launch adds some overhead.
• Statement cache is discarded when thread and connection is destroyed. Practically, with such context manager it doesn't work.

My practical use case is following: I create some pool of connections and since my journal_mode is WAL, often reads do not block each other and play well together.

Is it possible to expose connection start/stop as coroutines besides context manager?

Poll interval too big

Hi, thank you, this is a really useful library, however I think that this waiting time is too big and the responses are then too slow. Don't you want to consider lowering it e. g. hundred times?

SQLAlchemy support

Is it possible to get sqlalchemy support with the same interface as aiopg and aiomysql?

This would allow an easy way to use sqlite for local development or for an initial prototype, and then trivially upgrade to another DB.

Cannot set `row_factory` if I use the connection as a context manager

Description

It seems that I cannot set the row_factory attribute if I want to use the connection with async with later.

1. If I don't await the connection, I can't set row_factory on it, because it hasn't created an sqlite3.Connection object yet.
2. If I do await the connection, I can set row_factory on it, but if I then use the connection in an async with statement, Connection.__aenter__ will await on the aiosqlite.Connection object, which will start() it, which will start the thread twice, which leads to a nasty error.

As a workaround, I did this:

def connect(database: Union[str, Path], iter_chunk_size: int = 64, **kwargs: Any) -> aiosqlite.Connection:
    def connector() -> sqlite3.Connection:
        conn = sqlite3.connect(str(database), **kwargs)
        conn.row_factory = sqlite3.Row
        return conn
    return aiosqlite.Connection(connector, iter_chunk_size)

Maybe aiosqlite.connect should accept some flag or parameter to configure the row_factory, or maybe even a callback to do something when the connection is made?

Details

(don't think these matter)

• OS: Manjaro, 21.0.7
• Python version: 3.9.2
• aiosqlite version: 0.17.0
• Can you repro on 'main' branch? Yes
• Can you repro in a clean virtualenv? Yes

sleep tuning in _execute, please set to sleep(0.001)

Description

I tested various timing values on the test suite of our ORM using aiosqlite (https://github.com/Zeliboba5/tortoise-orm/)
By setting this value:
https://github.com/jreese/aiosqlite/blob/master/aiosqlite/core.py#L159

Details

This is the test run durations on my local system:
0.005 : Ran 101 tests in 7.051s (current value)
0.004 : Ran 101 tests in 6.149s
0.003 : Ran 101 tests in 4.830s
0.002 : Ran 101 tests in 3.741s
0.0015: Ran 101 tests in 3.635s
0.001 : Ran 101 tests in 2.446s
0.0005: Ran 101 tests in 2.538s
0.0001: Ran 101 tests in 2.530s

Conclusion

0.001 seems like a good wait time. I also suspect anything finer than a 1-ms resolution is likely to cause timing jitter and/or lock spinning.

This is on a Skylake i5-8300U Dell Latitude notebook with an aftermarket SATA SSD running linux.

Error importing

Description

>>> import aiosqlite Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<frozen importlib._bootstrap>", line 969, in _find_and_load File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 664, in _load_unlocked File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible File "/home/user/.local/lib/python3.5/site-packages/aiosqlite-0.8.1-py3.5.egg/aiosqlite/__init__.py", line 21, in <module> File "<frozen importlib._bootstrap>", line 969, in _find_and_load File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 664, in _load_unlocked File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible File "/home/user/.local/lib/python3.5/site-packages/aiosqlite-0.8.1-py3.5.egg/aiosqlite/core.py", line 110, in <module> File "/home/user/.local/lib/python3.5/site-packages/aiosqlite-0.8.1-py3.5.egg/aiosqlite/core.py", line 258, in Connection File "/usr/lib/python3.5/typing.py", line 649, in __getitem__ return Union[arg, type(None)] File "/usr/lib/python3.5/typing.py", line 552, in __getitem__ dict(self.__dict__), parameters, _root=True) File "/usr/lib/python3.5/typing.py", line 512, in __new__ for t2 in all_params - {t1} if not isinstance(t2, TypeVar)): File "/usr/lib/python3.5/typing.py", line 512, in <genexpr> for t2 in all_params - {t1} if not isinstance(t2, TypeVar)): File "/usr/lib/python3.5/typing.py", line 1077, in __subclasscheck__ if super().__subclasscheck__(cls): File "/usr/lib/python3.5/abc.py", line 225, in __subclasscheck__ for scls in cls.__subclasses__(): TypeError: descriptor '__subclasses__' of 'type' object needs an argument

Details

• OS: Ubuntu 16.04
• Python version: 3.5.2
• aiosqlite version: Latest
• Can you repro on master? -
• Can you repro in a clean virtualenv? I don't have venv

Support pysqlite3 for newer sqlite versions

Description

Currently aiosqlite relies upon importing the standard sqlite3 module from the python library. For many users, that is OK, but if you want to deploy your code on some platforms, the SQLite version installed by default cannot be easily upgraded through a package manager and some features are not available to these users (e.g. using WITHOUT ROWID in a table).

One avenue for fixing this issue is providing support for using pysqlite3 instead of the standard sqlite3 module. This can be done by either

1. Just using pysqlite3 by default
2. Update the setup script to add this feature as an optional installation requirement

For 2, I mean there should be the following two variations for installing aiosqlite

pip install aiosqlite
pip install aiosqlite[pysqlite3]

similar to how uvicorn gives the standard option while installing.

Details

• OS: Amazon Linux 2
• Python version: 3.7.9
• aiosqlite version: 0.16.0
• sqlite version installed: 3.7.17 but pysqlite3 supplies version 3.34.0
• Can you repro on 'main' branch? Yes
• Can you repro in a clean virtualenv? Yes

Context manager behaviour is different to the sqlite3 module

Description

A minor point that lead to a little confusion.

sqlite3's Connection's context manager is documented as handling committing and rolling back of transactions:

https://docs.python.org/3.9/library/sqlite3.html#using-the-connection-as-a-context-manager

import sqlite3

con = sqlite3.connect(":memory:")
con.execute("create table person (id integer primary key, firstname varchar unique)")

# Successful, con.commit() is called automatically afterwards
with con:
    con.execute("insert into person(firstname) values (?)", ("Joe",))

# Connection object used as context manager only commits or rollbacks transactions,
# so the connection object should be closed manually
con.close()

However aiosqlite seems to have the Connection context manager handle the opening and closing of the DB connection. In my usage, I notice that when exiting this manager, the transactions are not committed.

Possible solutions

I think at least some documentation about the current behaviour of the context managers would be a great improvement.

If we wanted to alter the behaviour, perhaps aisqlite.connect could return a new wrapper Connector class. Awaiting a Connector returns an active Connection, and async with manages closing it. Then the async with behaviour of the Connection itself is changed to handle transactions, as described.

Details

• OS: Debian 10 buster
• Python version: 3.7.3
• aiosqlite version: 0.17.0

How to return an active connection from a function?

Description

I'd like to return a configured connection from a function, something like this:

async def open_or_create():
    # prepare filesystem
    ...
    connection = aiosqlite.connect(filename)
    # create tables if database is empty
    async with connection.execute('PRAGMA user_version') as cursor:
        # prepare database
        ...
    # prepare connection
    ...
    return connection

Though apparently aiosqlite connections do not work outside of async with clause, and I get a ValueError (no active connection) exception on connection.execute. Moreover, adding async with to aiosqlite.connect line results in a broken connection getting returned to the caller, which gets RuntimeError (threads can only be started once) on its async with connection line.

Any chance to overcome this, am I missing something?

Details

• OS: Windows 10
• Python version: 3.6 (64-bit)
• aiosqlite version: 0.9.0

Python3.8 support?

Hi, just wondering what the status is of supporting python3.8? Github actions now have py3.8 available so you can at it to your build.yml

ContextManager is incorrectly typed

Description

ContextManager inherits from Coroutine, but is not type annotated, and does not match the Coroutine interface. We should make this better

Add support for pysqlite3

Duplicate

Why is aiosqlite so slower than sqlite3?

Description

Below is a code that inserts 1 million rows with sqlite3 library and prints its running time.

import time
import sqlite3

start = time.time()
def insert():
    with sqlite3.connect('numbers.db') as DB:
        DB.execute(f"BEGIN TRANSACTION;")
        for i in range(1000000):
            query = f"INSERT INTO numbers(number) VALUES ({i});"
            DB.execute(query)
        DB.commit()
insert()
end = time.time()
print(end-start)

If I run the code, the result is about 7 seconds on my computer. But if I use aiosqlite with the code below:

import time
import aiosqlite
import asyncio

start = time.time()
async def insert():
    async with aiosqlite.connect('numbers.db') as DB:
        await DB.execute(f"BEGIN TRANSACTION;")
        for i in range(1000000):
            query = f"INSERT INTO numbers(number) VALUES ({i});"
            await DB.execute(query)
        await DB.commit()
asyncio.run(insert())
end = time.time()
print(end-start)

then the result is about 110 seconds on my computer, which shows aiosqlite is so slower than sqlite3. Why is that?

Details

• OS: Linux
• Python version: 3.8
• aiosqlite version: 0.16.0
• Can you repro on 'main' branch? yes
• Can you repro in a clean virtualenv? yes

Request to release 0.7.1

Would be nice to have the following changes published with a new version:

• Avoid polling the Connection thread.
• Bugfix, exceptions were not safely set from the sqlite thread.

Cover Connection.backup

Description

Could you cover sqlite's backup? It came with python 3.7 and from what I understand you don't have it yet as I tried to do Connection.backup and it raised an AttributeError.

Details

• OS: Linux 5.7.7
• Python version: 3.8.3
• aiosqlite version: 0.13.0
• Can you repro on master?
• Can you repro in a clean virtualenv?

Connection attributes

Thanks for all the work!

If you don't mind, please add

db._conn.row_factory = aiosqlite.core.sqlite3.Row

into your example in readme, took me forever to figure it out :)

thank you

Use asyncio.Future instead of task ids and result queues

How to iterdump?

Description

I'm working with :memory: databases and I need a way to dump. Iterdump is perfect but I don't see it in the API. What to do or can you please add it?

Details

• OS: Linux
• Python version: 3.8
• aiosqlite version: 0.12.0
• Can you repro on master? yes
• Can you repro in a clean virtualenv? yes

Min dep version for typing-extensions

Description

typing_extensions>=3.7.2 is required to import Literal

Validate with

pip install pessimist==0.7.0
python -m pessimist . -p 1 -c 'make test && make lint'

(-p 1 because the tests clobber test.db and can't run concurrently)

If you add this to CI, also include --fast

Details

• OS: GNU/Linux
• Python version: 3.8.5
• aiosqlite version: main
• Can you repro on 'main' branch? yes
• Can you repro in a clean virtualenv? pretty sure

a item time 0.54s