I was not able to set min_size or max_size within Postgres about databases

[tomchristie]

Could you show me more precisely how to replicate the issue, eg...

1. The minimal possible source code that you can use to replicate?
2. The full traceback that is raised?

Will have a look at reopening the issue after that. Thanks!

[gbgduh]

Yeah, it seems to be the case. In the nutshell, the problem is that the passed db url to the pool is used further to acquire the connection, the connection isn't familiar with such query params in dsn and breaks. PostgresBackend._get_connection_kwargs parses that data from the url already and passes it as a dict, but it continues to stay in the url. It works if it's stripped from the urls after parsing.

I had similar problem from the other end though #69.
I would suggest to (at least for the pg back-end) pass query params as a kwargs dict not keeping them in the url, or something similar. But it's important to keep the db url compatible with sqlalchemy as well as it's also related to the `alembic.

Would be happy to drop any ideas in PR if it sounds reasonable.

[mths0x5f]

Sorry for the short issue report. I was kinda of sleepy. Now I'm just in doubt if missing something really obvious

I'm running Python 3.7.0 on macOS 10.13.6, installed with Homebrew I think

app.settings

DATABASE_URI = 'postgresql://postgres:postgres@localhost:5432/teste'
DATABASE_MIN_POOL_SIZE = 10
DATABASE_MAX_POOL_SIZE = 50

app.main

import databases
from fastapi import FastAPI
from app.settings import DATABASE_URI, DATABASE_MIN_POOL_SIZE, DATABASE_MAX_POOL_SIZE

app = FastAPI()

db = databases.Database(f'{DATABASE_URI}?min_size={DATABASE_MIN_POOL_SIZE}&max_size={DATABASE_MAX_POOL_SIZE}')


@app.on_event('startup')
async def startup():
    await db.connect()


@app.on_event('shutdown')
async def shutdown():
    await db.disconnect()

When run with uvicorn app.main:app, this is the full traceback I get:

INFO: Started server process [16477]
INFO: Waiting for application startup.
ERROR: Exception in 'lifespan' protocol
Traceback (most recent call last):
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/uvicorn/lifespan/on.py", line 43, in main
    await app_instance(self.receive, self.send)
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/starlette/routing.py", line 478, in asgi
    await self.startup()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/starlette/routing.py", line 461, in startup
    await handler()
  File "./app/main.py", line 17, in startup
    await db.connect()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/databases/core.py", line 59, in connect
    await self._backend.connect()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/databases/backends/postgres.py", line 58, in connect
    self._pool = await asyncpg.create_pool(str(self._database_url), **kwargs)
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 400, in _async__init__
    await self._initialize()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 417, in _initialize
    await first_ch.connect()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 125, in connect
    self._con = await self._pool._get_new_connection()
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/pool.py", line 463, in _get_new_connection
    **self._connect_kwargs)
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connection.py", line 1688, in connect
    max_cacheable_statement_size=max_cacheable_statement_size)
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 543, in _connect
    connection_class=connection_class)
  File "/Users/matheus/PycharmProjects/teste/venv/lib/python3.7/site-packages/asyncpg/connect_utils.py", line 519, in _connect_addr
    await asyncio.wait_for(connected, loop=loop, timeout=timeout)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/tasks.py", line 412, in wait_for
    return fut.result()
asyncpg.exceptions.UndefinedObjectError: unrecognized configuration parameter "min_size"
ERROR: Application startup failed. Exiting.

If I remove the parameters, it works. If I set ssl=True, it works either.

Thank you for taking a look into.

[mths0x5f]

I can confirm that that code raises the same exception run on Python 3.6.2 on Ubuntu 18.04.2 LTS.

[tomchristie]

Next thing to progress this would probably be to add a bunch of test cases to test_connection_options.py. Something like this...

@pytest.mark.parametrize("database_url", DATABASE_URLS)
@async_adapter
async def test_url_options(database_url):
    async with Database(database_url + '?min_size=5&max_size=10'):
        pass

@pytest.mark.parametrize("database_url", DATABASE_URLS)
@async_adapter
async def test_url_options(database_url):
    async with Database(database_url, min_size=5, max_size=10):
        pass

I'm not sure what beahvior we want with sqlite - we could have it silently ignore min_size, max_size and ssl.

[euri10]

I wrote a short test case to demonstrate the issue on a postgres backend,

from contextlib import suppress

POSTGRES_URLS = [url for url in DATABASE_URLS if urlparse(url).scheme =='postgresql']

urls_with_options = [
    (f"{POSTGRES_URLS[0]}?min_size=1&max_size=20", suppress()),
    (f"{POSTGRES_URLS[0]}?min_size=0&max_size=0", pytest.raises(ValueError)),
    (f"{POSTGRES_URLS[0]}?min_size=10&max_size=0", pytest.raises(ValueError)),
]
@pytest.mark.parametrize("database_url, expectation", urls_with_options)
@async_adapter
async def test_postgres_pool_size_connect(database_url, expectation):
    with expectation:
        async with Database(database_url) as db:
            db.connect()

A simple fix could be to strip off the url the query paramters in

databases/databases/backends/postgres.py

Line 65 in 88b381a

self._pool = await asyncpg.create_pool(str(self._database_url), **kwargs)

replacing the line with this
self._pool = await asyncpg.create_pool(str(urlparse(self._database_url._url)._replace(query=None).geturl()), **kwargs)

is that something I can send a PR for or you'd prefer to store on PostgresBackend a self._database_url_stripped that could have the base url without the query params and use that inside the connect() ?

[gvbgduh]

It might make the logic less explicit, so I think it's worth considering to do it in a way similar to other backends, like:

    async def connect(self) -> None:
        assert self._pool is None, "DatabaseBackend is already running"
        kwargs = self._get_connection_kwargs()
        self._pool = await aiomysql.create_pool(
            host=self._database_url.hostname,
            port=self._database_url.port or 3306,
            user=self._database_url.username or getpass.getuser(),
            password=self._database_url.password,
            db=self._database_url.database,
            autocommit=True,
            **kwargs,
        )

but with respect to the asyncpg create_pool and connect methods kwargs and execution order, as connect is called within the create_pool one.