Pytest Alembic Experiments

Experiments using pytest to validate Alembic database migrations.

Overview

This experiment tests:

whether alembic migrations can be tested under pytest, using pytest-alembic.
CRUD operations for SQLAlchemy models

There are two versions of these experiments:

version 1 uses a very simple, generic schema (single table with auto-incrementing ID and text label fields)
version 2 uses schemas from the BAS Locations Register project (multiple tables, PostGIS extension, FK relationship, function & trigger, enumerations)

Use version 1 if you want to see how this technique works. Use version 2 if you want to see how it works with a more complicated schema.

Setup

Requirements:

Git
Python
Poetry
Postgres with PostGIS extension

$ git clone https://github.com/felnne/pytest-alembic-experiments.git
$ poetry install
# this is not normally necessary, but for whatever reason Poetry won't recognise the local package and install it
$ echo ~/pytest-alembic-exp/src' > .venv/lib/python3.8/site-packages/coke.pth
$ psql -d postgres -c 'CREATE DATABASE pytest_alembic;'

Check things work manually:

$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run alembic upgrade head
$ SQLALCHEMY_SILENCE_UBER_WARNING=1 APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run python -m coke
$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run alembic downgrade base

Reset DB:

$ psql -d postgres -c 'DROP DATABASE IF EXISTS pytest_alembic;' && psql -d postgres -c 'CREATE DATABASE pytest_alembic;'

Check DB DSN environment variable set correctly:

$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run python -m coke.config

Usage

Locally:

$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic SQLALCHEMY_SILENCE_UBER_WARNING=1 poetry run pytest

CI:

commits to this repository will trigger GitLab CI to run tests using a service container, see .gitlab-ci.yml (if using GitLab)

Notes

once DSN removed from alembic.ini, all other config options are either static (package location) or defaults
offline mode is where Alembic generates SQL statements to run standalone, rather than Alembic modifying the DB
if DDL test fails, you can generate an automatic migration to see what's missing [1]
in CI, Alembic fails because the official PostGIS image includes more than just the PostGIS extension (tiger etc.)
- rather than create a new image etc. it was easiest to add psql in the app container and drop extra extensions
- see postgis/docker-postgis#187 for changing image to not add these extensions by default
testing alembic-utils resources:
- adding alembic-utils results in pytest-alembic detecting any replicable object in the DDL comparison test
- in our case, this includes the Function and Trigger used for updating the updated_at column (which we'd want to track), but also includes objects are things like the PostGIS extension, it's indexes and views (which we don't)
- I tried to replicate all of these objects into models.py using the alembic-utils classes, and then use register_entities([...]) referencing these objects
- this worked in that I think in the pytest-alembic DDL test, it was trying to create these entities as part of the comparison, but was failing because the resources already existed (as they are created when enabling the PostGIS extension)
- to fix that, you would need to use CREATE OR REPLACE VIEW or CREATE VIEW IF NOT EXISTS etc. to fail gracefully
- within the alembic-util classes, there is logic to do this, but I couldn't see how to trigger it
- at the moment, testing these resources isn't a strict requirement, so I have instead excluded them from the comparison using alembic's exclude logic
- I think with some further experimentation and hacking this could probably be made to work so worth adding as a backlog item
pytest-alembic experimental tests:
- reviewing the two experimental tests ('all models register on metadata' and 'downgrade leaves no trace')
- both would likely fail due to the workarounds used to exclude replaceable objects
- given these tests are experimental, I don't think this is a significant problem
PostGIS spatial references table:
- as a late edition, having excluded all replaceable entities as part of integrating alembic-utils, I configured alembic to exclude the PostGIS spatial references table as well, rather than needing this to be defined as an app model
- this is compatible with the pytest-alembic DDL test

[1]

$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run alembic upgrade head
$ APP_DB_DSN=postgresql://$user@localhost/pytest_alembic poetry run alembic revision --autogenerate

If SQLAlchemy models are not as up to date as the Alembic models, the upgrade/downgrade steps will be flipped around.

TODO

SQLAlchemy tests
Does the rolled back in the DB fixture mean we don't need migrations to be run, as it doesn't touch the DB?
- no, committing the session modifies the DB
what is 'offline mode' in Alembic?
- where Alembic generates SQL to run externally
DSN is defined multiple times [1]
GitLab CI
More realistic tests (i.e. from locations register)
review Pytest-alembic experimental tests

Licence

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

felnne / pytest-alembic-experiments Goto Github PK