Comments (7)
Could you show me more precisely how to replicate the issue, eg...
- The minimal possible source code that you can use to replicate?
- The full traceback that is raised?
Will have a look at reopening the issue after that. Thanks!
from databases.
Yeah, it seems to be the case. In the nutshell, the problem is that the passed db url to the pool is used further to acquire the connection, the connection isn't familiar with such query params in dsn and breaks. PostgresBackend._get_connection_kwargs
parses that data from the url already and passes it as a dict, but it continues to stay in the url. It works if it's stripped from the urls after parsing.
I had similar problem from the other end though #69.
I would suggest to (at least for the pg back-end) pass query params as a kwargs dict not keeping them in the url, or something similar. But it's important to keep the db url compatible with sqlalchemy
as well as it's also related to the `alembic.
Would be happy to drop any ideas in PR if it sounds reasonable.
from databases.
Sorry for the short issue report. I was kinda of sleepy. Now I'm just in doubt if missing something really obvious
I'm running Python 3.7.0 on macOS 10.13.6, installed with Homebrew I think
app.settings
DATABASE_URI = 'postgresql://postgres:postgres@localhost:5432/teste'
DATABASE_MIN_POOL_SIZE = 10
DATABASE_MAX_POOL_SIZE = 50
app.main
import databases
from fastapi import FastAPI
from app.settings import DATABASE_URI, DATABASE_MIN_POOL_SIZE, DATABASE_MAX_POOL_SIZE
app = FastAPI()
db = databases.Database(f'{DATABASE_URI}?min_size={DATABASE_MIN_POOL_SIZE}&max_size={DATABASE_MAX_POOL_SIZE}')
@app.on_event('startup')
async def startup():
await db.connect()
@app.on_event('shutdown')
async def shutdown():
await db.disconnect()
When run with uvicorn app.main:app
, this is the full traceback I get:
INFO: Started server process [16477]
INFO: Waiting for application startup.
ERROR: Exception in 'lifespan' protocol
Traceback (most recent call last):
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 43, in main
await app_instance(self.receive, self.send)
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/starlette/routing.py", line 478, in asgi
await self.startup()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/starlette/routing.py", line 461, in startup
await handler()
File "./app/main.py", line 17, in startup
await db.connect()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/databases/core.py", line 59, in connect
await self._backend.connect()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/databases/backends/postgres.py", line 58, in connect
self._pool = await asyncpg.create_pool(str(self._database_url), **kwargs)
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 400, in _async__init__
await self._initialize()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 417, in _initialize
await first_ch.connect()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 125, in connect
self._con = await self._pool._get_new_connection()
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 463, in _get_new_connection
**self._connect_kwargs)
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connection.py", line 1688, in connect
max_cacheable_statement_size=max_cacheable_statement_size)
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 543, in _connect
connection_class=connection_class)
File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 519, in _connect_addr
await asyncio.wait_for(connected, loop=loop, timeout=timeout)
File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 412, in wait_for
return fut.result()
asyncpg.exceptions.UndefinedObjectError: unrecognized configuration parameter "min_size"
ERROR: Application startup failed. Exiting.
If I remove the parameters, it works. If I set ssl=True, it works either.
Thank you for taking a look into.
from databases.
I can confirm that that code raises the same exception run on Python 3.6.2 on Ubuntu 18.04.2 LTS.
from databases.
Next thing to progress this would probably be to add a bunch of test cases to test_connection_options.py
. Something like this...
@pytest.mark.parametrize("database_url", DATABASE_URLS)
@async_adapter
async def test_url_options(database_url):
async with Database(database_url + '?min_size=5&max_size=10'):
pass
@pytest.mark.parametrize("database_url", DATABASE_URLS)
@async_adapter
async def test_url_options(database_url):
async with Database(database_url, min_size=5, max_size=10):
pass
I'm not sure what beahvior we want with sqlite
- we could have it silently ignore min_size
, max_size
and ssl
.
from databases.
I wrote a short test case to demonstrate the issue on a postgres backend,
from contextlib import suppress
POSTGRES_URLS = [url for url in DATABASE_URLS if urlparse(url).scheme =='postgresql']
urls_with_options = [
(f"{POSTGRES_URLS[0]}?min_size=1&max_size=20", suppress()),
(f"{POSTGRES_URLS[0]}?min_size=0&max_size=0", pytest.raises(ValueError)),
(f"{POSTGRES_URLS[0]}?min_size=10&max_size=0", pytest.raises(ValueError)),
]
@pytest.mark.parametrize("database_url, expectation", urls_with_options)
@async_adapter
async def test_postgres_pool_size_connect(database_url, expectation):
with expectation:
async with Database(database_url) as db:
db.connect()
A simple fix could be to strip off the url the query paramters in
databases/databases/backends/postgres.py
Line 65 in 88b381a
replacing the line with this
self._pool = await asyncpg.create_pool(str(urlparse(self._database_url._url)._replace(query=None).geturl()), **kwargs)
is that something I can send a PR for or you'd prefer to store on PostgresBackend a self._database_url_stripped
that could have the base url without the query params and use that inside the connect() ?
from databases.
It might make the logic less explicit, so I think it's worth considering to do it in a way similar to other backends, like:
async def connect(self) -> None:
assert self._pool is None, "DatabaseBackend is already running"
kwargs = self._get_connection_kwargs()
self._pool = await aiomysql.create_pool(
host=self._database_url.hostname,
port=self._database_url.port or 3306,
user=self._database_url.username or getpass.getuser(),
password=self._database_url.password,
db=self._database_url.database,
autocommit=True,
**kwargs,
)
but with respect to the asyncpg create_pool and connect methods kwargs and execution order, as connect
is called within the create_pool
one.
from databases.
Related Issues (20)
- Unable to retrieve foreignkey relation object from select statement HOT 2
- Password containing `/` cannot be used (MySQL)
- Support for UNIX domain socket for MySQL
- No module named 'asyncmy.connection' on python 3.11.3
- Warn for open transactions when a connection is garbage collected
- Unable to connect to database in docker
- MySQL Connection Pool Doesn't Seem to Work HOT 3
- Drop support for python 3.7 HOT 1
- New test `test_should_remove_ref_on_disconnect` in 0.8.0 fails on Fedora Linux HOT 10
- `IndexError` on force_rollback of Transaction in tests. HOT 13
- ImportError Postgresql HOT 1
- Transaction and gather HOT 2
- session manager/ session maker
- Allow passing string values for ssl query string in PostgreSQL URL HOT 1
- Version not match (0.8.0) depends on sqlalchemy (>=1.4.42,<1.5) HOT 2
- mypy's "has no attribute" error
- create table does not create unique keys and indexes
- new 0.8 version for sqlite connection with '?mode=ro' not working, Error can't open database
- Caching disabled in sqlalchemy queries HOT 1
- Potential Inefficiency in Connection Management Leading to "Too Many Connections" Error HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from databases.