aio-libs / aioodbc Goto Github PK

ref:
[1] https://github.com/mkleehammer/pyodbc/wiki/Using-an-Output-Converter-function
[2] https://github.com/mkleehammer/pyodbc/wiki/Unicode

add tests on windows with MSSQL using
http://www.appveyor.com/docs/services-databases

Alternate title: "aioodbc does not support PEP 561"

PEP 561 – Distributing and Packaging Type Information defines the mechanisms for distributing packages with type hints:

Package maintainers who wish to support type checking of their code MUST add a marker file named py.typed to their package supporting typing. This marker applies recursively: if a top-level package includes it, all its sub-packages MUST support type checking as well.

Since aioodbc does not contain a py.typed marker file, developers may find themselves without any type hints for the package. Developers are unable to take advantage of the type annotations already part of the project.

Most typed packages on PyPI publish place their py.typed file in the namespace directory (e.g., "<root>/aioodbc") to better centralize the type checking to the package source code. It would be a huge improvement to the developer experience if the aioodbc project included a py.typed marker file alongside the package.

aioodbc requires at least pyodbc version 5 which does not support Python 3.7 anymore.
Therefore the "python-requires" should be changed to ">=3.8"

Code:

async with glob.mysq_pool.acquire() as conn:
    async with conn.cursor() as cur:
        await cur.execute(sql, params)

        r = await cur.fetchall()
        for element in r:
            print(cur.description)
            print(element)

    await conn.close()

output:

(('Id', <class 'int'>, None, 10, 10, 0, True), ('GuildId', <class 'int'>, None, 20, 20, 0, False), ('ChannelId', <class 'int'>, None, 20, 20, 0, False), ('Type', <class 'int'>, None, 3, 3, 0, False), ('Format', <class 'str'>, None, 100, 100, 0, False))
(9319, 611111746538700801, 668929958751764501, 1, '? Online: %')

I received "? Online: %" instead of "? Online: %s", but database contains full value.

Database: MariaDB 10.3.18
Driver: MySQL ODBC 8.0 Unicode Driver

For my API, I send a few alter session statements on connect.

I would like to be able to specify an on connect hook such that recycled connections run said hook.

I would love to submit a PR for this as I'm currently monkey patching around this.

Is this project still fully active? I haven't seen activity in a while.

Can you please add support for Cursor.cancel()?

pyodbc supports it https://github.com/mkleehammer/pyodbc/blob/45395fe723b837e836643897ff3df57034c05b22/src/cursor.cpp#L2034

would it be possible to have a wheel on pypi ? like aioodbc-0.3.3-py3-none-any.whl ?

Hi! I have some problem.
I use on linux system:

• unixODBC-2.3.4;
• mysql-5.6.35;
• myodbc-5.2.7;
• python-3.6.0;
• pyodbc-4.0.11;
• aioodbc-0.0.3.

For any query I get an error message, not exception:

Error in my_thread_global_end(): 2 threads didn't exit

Query is executed correctly, only error message on console.

Code is very simple, from example, with pool and without pool:

dsn = f'Driver=myodbc;User={USER};Password={PASSWORD};Database={DB};'

async with aioodbc.create_pool(dsn=dsn, loop=loop) as pool:
    async with pool.acquire() as conn:
        async with conn.cursor() as cur:
            await cur.execute('SHOW TABLES;')
            val = await cur.fetchone()

Same code with just pyodbc runs without error message:

import pyodbc

cnxn = pyodbc.connect(dsn)
cursor = cnxn.cursor()
cursor.execute('SHOW TABLES;')
r = cursor.fetchall()

MySQL settings is default. Why can this error message?

Hello, I'm attempting to connect to an in-memory SQLite database in the following manner.

import asyncio
import aioodbc

DB_CONNSTR = "Driver=SQLite;Database=:memory:"

async def run_db_loop(q):
    conn = await aioodbc.connect(dsn=DB_CONNSTR, loop=loop)

loop.run_until_complete(run_db_loop(queue.async_q))

This produces the following error on macOS Sierra v10.12.6:

~/.pyenv/versions/3.6.2/lib/python3.6/site-packages/aioodbc/connection.py in _connect(dsn, autocommit, ansi, timeout, loop, executor, echo, **kwargs)
     43                       timeout=timeout, echo=echo, loop=loop, executor=executor,
     44                       **kwargs)
---> 45     await conn._connect()
     46     return conn
     47 

~/.pyenv/versions/3.6.2/lib/python3.6/site-packages/aioodbc/connection.py in _connect(self)
     81                           timeout=self._timeout,
     82                           **self._kwargs)
---> 83         self._conn = await f
     84 
     85     @property

~/.pyenv/versions/3.6.2/lib/python3.6/concurrent/futures/thread.py in run(self)
     53 
     54         try:
---> 55             result = self.fn(*self.args, **self.kwargs)
     56         except BaseException as e:
     57             self.future.set_exception(e)

Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'SQLite' : file not found (0) (SQLDriverConnect)")

Prior to running the script I performed brew install unixodbc.

Any idea what might be going on here?

I've setup a connection pool which automatically starts with 10 connections. How can I set autocommit=False on the acquired connection?

The upstream issue is: mkleehammer/pyodbc#341

Would it be possible to pin the pyodbc version? That way reinstalling dependent projects would be reliable and if a specific version is needed it can be overridden per project (and maintain relaibility).

And thanks for this project! The world doesn't have enough async database clients.

Make sure issue described in #79 gone away with new version of MySQL and libmyodbc.

https://dev.mysql.com/downloads/connector/odbc/
http://askubuntu.com/questions/800216/installing-ubuntu-16-04-lts-how-to-install-odbc

When using aioodbc with a connection pool under Python 3.9, the following error is reported:

  File "/usr/local/lib/python3.9/site-packages/aioodbc/pool.py", line 28, in _create_pool
    with (await pool._cond):
TypeError: object Condition can't be used in 'await' expression

The affected code is:

According to the official documentation, the asyncio.Condition object should be handled using an async with construct instead of a with with an await expression as done in aiodbc.

I will try to send a pull request fixing this soon but I don't have much time nowadays. I also hope that this project is not abandoned either as it is very useful.

Do like this:

REPO_NAME={{ YOUR_REPO_NAME }}
travis encrypt -r "aio-libs/${REPO_NAME}" --api-endpoint 'https://api.travis-ci.com/'

Ref: https://github.com/orgs/aio-libs/teams/admins/discussions/9

aioodbc's last release seems to be Mar 16, 2023, seven months ago. however fix for #289 was made one day later and is not in this release. please release all current bugfixes thanks!

Apologies if this is not the place to ask, as I do not use Github often, but is this package dead?

It looks like there are ~20 open issues, some pretty important, and the last closed issue was in 12/2020.

Pool.acquire returns a _PoolConnectionContextManager.

After __aenter__ a pyodbc.OperationalError: ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (0) (SQLExecDirectW)') can be thrown in which case the connection may be permanently bad.

We now go to its __aexit__ where it unconditionally calls await self._pool.release(self._conn).

Pool.release at this point doesn't know if the connection is bad.

Instead if pyodbc.OperationalError is raised the connection should probably always be closed.

Hello

Will aioodbc support fast_executemany method ?

Hello. How can i create a permanent connection to database with aiodbc and use it for all queries like this:

cnxn   = pyodbc.connect(...)
cursor = cnxn.cursor()
cursor.execute(...)

I don't want to create a new connection to the database every time to make a query.
Thanks.

Request

New release with 3.12 wheels in it.

Issue

pip install aioodbc fails on Python 3.12.

Dependabot updates have upgraded several packages that do not work with Python 3.6 (via Debian Jessie). Upgrading the test environment to use Debian Buster resolves the issue.

I'm trying to build an FastAPI application with using MSSQL, I couldn't find any ORM library supports aioodbc. Is there any library available ?

Python 3.10 changed the signature of the asyncio.Condition() class :

Changed in version 3.10: Removed the loop parameter.

The current aioodbc.Pool constructor uses the old signature which raises the TypeError: Condition.__init__() got an unexpected keyword argument 'loop' error.

How to reproduce

With Python 3.10 or 3.11, run the following code

import asyncio
import aioodbc


loop = asyncio.get_event_loop()


async def example_pool():
    dsn = 'Driver=SQLite;Database=sqlite.db'
    pool = await aioodbc.create_pool(dsn=dsn, loop=loop)

    async with pool.acquire() as conn:
        cur = await conn.cursor()
        await cur.execute("SELECT 42;")
        r = await cur.fetchall()
        print(r)
        await cur.close()
        await conn.close()
    pool.close()
    await pool.wait_closed()

if __name__ == '__main__':
    loop.run_until_complete(example_pool())

Actual result

tmp.py:5: DeprecationWarning: There is no current event loop
  loop = asyncio.get_event_loop()
Traceback (most recent call last):
  File "tmp.py", line 23, in <module>
    loop.run_until_complete(example_pool())
  File "C:\Program Files\Python311\Lib\asyncio\base_events.py", line 650, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "tmp.py", line 10, in example_pool
    pool = await aioodbc.create_pool(dsn=dsn, loop=loop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Lib\site-packages\aioodbc\pool.py", line 25, in _create_pool
    pool = Pool(minsize=minsize, maxsize=maxsize, echo=echo, loop=loop,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\Lib\site-packages\aioodbc\pool.py", line 47, in __init__
    self._cond = asyncio.Condition(loop=loop)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Condition.__init__() got an unexpected keyword argument 'loop'

Process finished with exit code 1

Expected result

no error

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

Passed the parameter echo=True to the pool, but cursor.echo == False.

pool = await aioodbc.create_pool(
    dsn=self._dsn, minsize=1, maxsize=self._pool_size,
    loop=self._loop, echo=True, **self._kwargs)
async with pool.acquire() as connection:
    async with connection.cursor() as cursor:
        assert cursor.echo == True

I cannot connect MS SQL server using aioodbc.

Here is my code:

import asyncio
import aioodbc

loop = asyncio.get_event_loop()
async def connect_db():
    dsn = 'Driver=ODBC Driver 11 for SQL Server;Server=123.456.789.55,1000/mycompany;\
           Database=Asia_client;User=myname;\
           Password=mypass'
    conn = await aioodbc.connect(dsn=dsn, loop=loop)

    cur = await conn.cursor()
    await cur.execute("SELECT top 100 *;")
    rows = await cur.fetchall()
    await cur.close()
    await conn.close()

loop.run_until_complete(connect_db())

pyodbc.Error: ('08001', '[08001] [Microsoft][ODBC Driver 11 for SQL Server]SQL Server Network Interfaces: Connection string is not valid [87].  (87) (SQLDriverConnect)')

On the other and,
It is fine to connect using sqlalchemy:

engine = sqlalchemy.create_engine("mssql+pyodbc://myname:[email protected],1000/mycompany?driver=ODBC+Driver+11+for+SQL+Server")

How to solve the problem?
Thank you very much.

Python 3.6
aioodbc 0.1.0
Windows
SQLite

The following just can't work

async def example():
    async aioodbc.connect(...) as conn, conn.cursor() as cursor:
        async for row in await cursor.execute("SELECT * FROM some_table"):
            print(row)

Currently for it to work I have to use two lines

async def example():
    async aioodbc.connect(...) as conn, conn.cursor() as cursor:
        cursor.execute("SELECT * FROM some_table")
        async for row in await cursor:
            print(row)

or not use async for loop but that defeats the purpose of async.

Hi,

What about SQLAlchemy integration like in aiopg? Do you have any plans? Or it's already supported but missed in docs?

Probably following image could be used:
https://hub.docker.com/r/wnameless/oracle-xe-11g/

I'm using aioodbc to talk to MySQL. executemany works for me in inserting many rows into a table. Hoping to increase throughput, I tried to async call executemany concurrently like this:

async for x in data:
    task = asyncio.ensure_future(cursor.executemany(...))
    await queue.put(task)

The queue has capacity 10. This leads to Segmentation fault w/o any additional info. I guess this has to do with the interaction between async and threads, but don't know for sure. Any idea why this is happening? Thanks.

The default value is meant to be cursor.arraysize.

https://www.python.org/dev/peps/pep-0249/#fetchmany

The psycopg implementation here may be of use.

https://github.com/psycopg/psycopg/blob/master/psycopg/psycopg/cursor.py#L607-L626

Hello,

I have a table with few thousands entries which could be represented with these few lines:

ID;Column1;Column2;Column3
1;"line1";"CompanyA";""
2;"line2";"CompanyA";"someText"
3;"line3";"CompanyA";""

I am using the HFSQL driver to retrieve information from a database with the same name.

By executing the following code, I got wrong information:

  sql_request = 'SELECT id, Column1, Column2, Column3 FROM TABLE WHERE Column2 in ('CompanyA') LIMIT 400'
  await cursor.execute(sql_request)
  rows = []
  while row := await cursor.fetchone():
      rows.append(self.obj_helper(row))
  await cursor.close()
  return rows

Result:
1;"line1";"CompanyA"; ""
2;"line2";"CompanyA";"someText"
3;"line3";"CompanyA";"someText"

Where can I dig to understand why, after a line with a correct string, a row without value for one of its column takes the previous entry ?
Thank you,
R

Hey,
I have confusing problem. There is an endpoint that makes for example 15 queries. On development environment I create pool of exactly 10 to 20 connections and I expect that after hitting this endpoint I will be able to hit it again, especially that it don't use those connections all at once, but I get concurrent.futures._base.CancelledError error instead so it looks like connection didn't return to pool and framework triggered standard RequestTimeoutError (I made this assumption because increasing pool size solves the problem).

I make queries with this code:

from collections import namedtuple

from sanic.exceptions import NotFound


def _row_namedtuple(cursor):
    return namedtuple('Row', _field_names(cursor))


def _field_names(cursor):
    return [name[0].lower() for name in cursor.description]


class DbExecutor:
    def __init__(self, sql, pool, params=()):
        self.sql = sql
        self.params = params
        self.pool = pool
        self._connection = None
        self._cursor = None

    async def __aenter__(self):
        self._connection = await self.pool.acquire()
        self._cursor = await self._connection.cursor()
        await self._cursor.execute(self.sql, self.params)
        return self._cursor

    async def __aexit__(self, exc_type, exc, tb):
        await self._cursor.close()
        await self._connection.close()


async def fetch_dict_all(sql, pool, params=()):
    async with DbExecutor(sql, pool, params) as cursor:
        return [dict(zip(_field_names(cursor), row)) for row in await cursor.fetchall()]


async def fetch_dict_row(sql, pool, params=()):
    async with DbExecutor(sql, pool, params) as cursor:
        row = await cursor.fetchone()
        return dict(zip(_field_names(cursor), row)) if row else []

create pool with this code:

@app.listener('before_server_start')
async def create_pool(app, loop):
    dsn = 'DRIVER=OracleODBC;DBQ=server;UID=login;PWD=password'
    app.db = await aioodbc.create_pool(dsn=dsn, loop=loop, minsize=10, maxsize=10)


@app.listener('after_server_stop')
async def discard_pool(app, loop):
    app.db.close()
    await app.db.wait_closed()

example of query:

details = await fetch_dict_all('sql query', app.db, params)

Ubuntu 16.04
Oracle ODBC Driver
aioodbc == 0.2.0
pyodbc == 4.0.21

this method is for some reason not implemented

This method is extremely important for pyodbc, here's the docs for how it is used with pyodbc , here are github issues where setinputsizes is the only way to make things work: mkleehammer/pyodbc#1239 mkleehammer/pyodbc#1221 mkleehammer/pyodbc#1021 many more

SQLAlchemy will work around for now by hitting impl directly

cc @gordthompson

I'm using SQL Server 2019 with aioodbc I'm observing a situation wherein when the connection becomes stale the I'm getting exception when try to execute any query. Currently there is no check for validating connection when we acquire connection from pool. Check for connection validity happens when there is error in statement execution. When statement gets errored out the caller also gets exception. The logic is falling apart for my long running python server. Currently I've a pool size of 25 so I've to wait for 25 requests to fail so that connections get closed in pool. The 26th request will get a fresh connection.

I think the behavior of acquire should run a validation query on connection object if connection validation is failed then it should remove connection from pool and try to obtain another valid one.

I'm trying to use aioodbc for working with SQLite database and struggling with issue that database content is not preserved after closing connection.

Take a look at the following example:

import asyncio
import aioodbc


loop = asyncio.get_event_loop()


async def test_example():
    dsn = 'Driver=SQLite;Database=test_db.sqlite'
    async with aioodbc.connect(dsn=dsn, loop=loop) as conn:
        async with await conn.cursor() as cur:
            await cur.execute("CREATE TABLE test (id INTEGER PRIMARY KEY)")
            await cur.execute("INSERT INTO test VALUES (1)")
            await cur.execute("SELECT * FROM test")
            r = await cur.fetchall()
            print(r)

    async with aioodbc.connect(dsn=dsn, loop=loop) as conn:
        async with await conn.cursor() as cur:
            # This line fails:
            # pyodbc.Error: ('HY000', '[HY000] [SQLite]no such table: test (1) (1) (SQLExecDirectW)')
            await cur.execute("SELECT * FROM test")
            r = await cur.fetchall()
            print(r)

loop.run_until_complete(test_example())

in this example I create table, insert value, close DB, then reopen DB and try to read inserted value. This fails with:

$ python test_odbc.py 
[(1, )]
Traceback (most recent call last):
  File "test_odbc.py", line 24, in <module>
    loop.run_until_complete(test_example())
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test_odbc.py", line 20, in test_example
    await cur.execute("SELECT * FROM test")
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
pyodbc.Error: ('HY000', '[HY000] [SQLite]no such table: test (1) (1) (SQLExecDirectW)')

Perhaps I'm missing some configuration parameter?
test_db.sqlite file is created, but empty.

I'm running on Ubuntu 16.04 with Python 3.5 in virtualenv.

$ pip list
aioodbc (0.0.3)
pip (8.1.1)
pkg-resources (0.0.0)
pyodbc (3.0.10)
setuptools (20.7.0)

pyodbc's autocommit field can be set to change the autocommit state of the connection while it is open. Updating the property to allow the value to be set brings the implementation more in line with implementing the "same api" as pyodbc.

Is it possible to add type annotations to aioodbc so that mypy properly determine the types of objects being imported from aioodbc?