boustrophedon / pgtemp

Rust library and daemon for easily starting postgres databases per-test without Docker

pgtemp

pgtemp is a Rust library and cli tool that allows you to easily create temporary PostgreSQL servers for testing without using Docker.

The pgtemp Rust library allows you to spawn a PostgreSQL server in a temporary directory and get back a full connection URI with the host, port, username, and password.

The pgtemp cli tool allows you to even more simply make temporary connections, and works with any language: Run pgtemp and then use its connection URI when connecting to the database in your tests. pgtemp will then spawn a new postgresql process for each connection it receives and transparently proxy everything over that connection to the temporary database. Note that this means when you make multiple connections in a single test, changes made in one connection will not be visible in the other connections, unless you are using pgtemp's --single mode.

pgtemp supports loading (and dumping, in the library) the database to/from dumpfiles via pg_dump.

Note that the default postgres authentication configuration (pg_hba.conf) in most cases allows all local connections. Since pgtemp only allows you to make servers that listen on localhost, this means in most cases you do not need to provide a password to connect. You may set the server's hba_file parameter in PgTempDBBuilder::with_config_param or use the pgtemp daemon's -o flag to pass hba_file there.

Requirements

You must install both the postgresql client and server packages. On Debian/Ubuntu, they are postgresql postgresql-client, on Fedora they are postgresql postgresql-server, and on Arch Linux they are postgresql postgresql-libs. Note also that Debian/Ubuntu install the standard postgres binaries into their own directory, so you must add them to your path. For an Ubuntu GitHub Actions runner, it looks like:

steps:
  - name: Install postgres
    run: sudo apt-get install postgresql postgresql-client
  - name: Update path
    run: find /usr/lib/postgresql/ -type d -name "bin" >> $GITHUB_PATH

To install the CLI tool, you must install it with the --features cli or --all-features options

cargo install pgtemp --features cli

Design

pgtemp is a fairly simple program and there are other existing libraries like testing.postgresql for Python and pgtest for Go that all work the same way:

• Do some setup, like creating temporary directories and copy files
• Run the initdb program
• Start the postgres server
• Wait until postgres has started

The novel idea (as far as I'm aware, although I also only found out about the above python/go libraries after coming up with the initial library idea) in pgtemp is the CLI/daemon which automatically provides connections to new temporary databases upon each connection.

Examples

CLI

$ cargo install --all-features pgtemp
# username, password, port, and database name are all configurable based on the provided connection URI
$ pgtemp postgresql://localhost:6543/mytestdb 
starting pgtemp server at postgresql://postgres:password@localhost:6543/mytestdb
$ psql postgresql://postgres:password@localhost:6543/mytestdb
psql (16.1)
Type "help" for help.

postgres=#

See examples/ directory for examples:

• A python example with sqlalchemy and alembic, demonstrating usage with the pgtemp cli's normal and single modes

Library

use pgtemp::PgTempDB;
use sqlx::postgres::PgConnection;
use sqlx::prelude::*;

#[tokio::test]
fn cool_db_test() {
    let db = PgTempDB::async_new().await;
    let mut conn = sqlx::postgres::PgConnection::connect(&db.connection_uri())
        .await
        .expect("failed to connect to temp db");

    // ... do the rest of your test

    // db is shut down and files cleaned up upon drop at the end of the test
}

Examples:

• A simple diesel example with axum
• A more complicated "task queue" example using triggers and LISTEN/NOTIFY with sqlx and axum

See the tests/ directory for complete library usage.

pgtemp's Issues

Can't run in Github Actions due to `sudo` use

When I try to use this in Github Actions I get these errors:

initdb failed! stdout: 

stderr: sudo: initdb: command not found
), location: Location { file: "/root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pgtemp-0.3.0/src/run_db.rs", line: 72, col: 9 }

from what I see, this happens because sudo can't find initdb in its secure path - adding the path to postgres to $PATH doesn't solve the issue.
I mitigated the issue by copying the whole postgresql bin &lib folders into /usr/bin & /usr/lib respectively, but it would be nice if the library could handle this.

Shutdown (without persist) hangs in Github actions

I can't repro this locally, so I don't know what the source of the issue is. The temporary DB, created with PgTempDB::async_new().await (so persist isn't set), hangs for minutes after a call to shutdown. Maybe there are some existing connections to the DB, but according to the documentation, this shouldn't block shutdown when persist isn't set.

This happens on a runner running Ubuntu noble, with postgresql-16 installed.

Failure to set up database due to race between Postgres startup and `createdb`

Hi, thanks for writing this, I was thinking of such a thing months ago and it's always nice to see someone else wrote what I wanted :D.

The only problem I'm having is that occasionally the tests fail to start up. This is much more reproducible when my backup software is running in the background, putting a load on the system (and disk?).

The output I get most often is as follows:

createdb failed! stdout: 

stderr: createdb: error: connection to server at "localhost" (127.0.0.1), port 41499 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?

thread 'tests::test_oidc_auth_flow::test_userinfo_bad_auth' panicked at /home/rei/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pgtemp-0.3.0/src/lib.rs:378:14:
failed to start pgtemp server: JoinError::Panic(Id(2), ...)

Occasionally I get the following instead

stderr: createdb: error: connection to server at "localhost" (127.0.0.1), port 35153 failed: FATAL:  the database system is starting up

I think these errors mean that Postgres hasn't started up yet, but pgtemp is already trying to run createdb.

I'm not sure if there's a perfect way of waiting for startup to happen (I guess you can possibly parse the logs?), so perhaps just trying createdb a few times in a loop with a small delay is the way to go.

I'm using pgtemp v0.3.0.