theory / pgenv

PostgreSQL binary manager

pgenv - PostgreSQL binary manager

Synopsis

pgenv help

# Check dependencies
pgenv check

# Show versions available to build
pgenv available

# Build PostgreSQL server
pgenv build 10.4

# Switch PostgreSQL version
pgenv switch 10.4

# Use PostgreSQL version
pgenv use 10.4

# Stop current version
pgenv stop

# Start current version
pgenv start

# Restart current version
pgenv restart

# Show current version
pgenv version

# List built versions
pgenv versions

# Clear current version
pgenv clear

# Remove PostgreSQL version
pgenv remove 8.0.25

Description

pgenv is a simple utility to build and run different releases of PostgreSQL. This makes it easy to switch between versions when testing applications for compatibility.

Installation

1. Check out pgenv into ~/.pgenv.

   git clone https://github.com/theory/pgenv.git ~/.pgenv

2. Add ~/.pgenv/bin and ~/.pgenv/pgsql/bin to your $PATH for access to the pgenv command-line utility and all the programs provided by PostgreSQL:

   echo 'export PATH="$HOME/.pgenv/bin:$HOME/.pgenv/pgsql/bin:$PATH"' >> ~/.bash_profile

   Ubuntu note: Modify your ~/.profile instead of ~/.bash_profile.

   Zsh note: Modify your ~/.zshrc file instead of ~/.bash_profile.

3. Restart your shell as a login shell so the path changes take effect. You can now begin using pgenv.

   exec $SHELL -l

4. Build a version of PostgreSQL:

   pgenv build 10.4

Configuration

By default, all versions of PostgreSQL will be built in the root of the project directory (generally in ~/.pgenv.). If you'd like them to live elsewhere, set the $PGENV_ROOT environment variable to the appropriate directory.

It is possible to configure which programs, flags and languages to build using a configuration file before the program launches the build. For a more detailed configuration, see the pgenv config command below.

You can use a local PostgreSQL git repo instead of downloading tarballs for the build step by setting the external $PGENV_LOCAL_POSTGRESQL_REPO environment variable to the appropriate absolute path.

Running scripts

It is possible to run custom script when particular events happen. The purpose is to let the user to configure the cluster as much as she want, for instance installing extension or pre-loading data.

Custom scripts are driven by a set of configuration variables as follows:

• PGENV_SCRIPT_POSTINSTALL is an executable script run as soon as the build finishes. The return value of the script does not affect the pgenv workflow. The script gets the version of PostgreSQL being installed as first argument.

• PGENV_SCRIPT_POSTINITDB is an executable script run as soon as the use command ends the initdb phase, that happens only the PGDATA has not been initialized (i.e., on very first use). The return value of the script does not affect the pgenv workflow. The script gets the current PGDATA path as first argument.

• PGENV_SCRIPT_POSTSTART is an executable script executed each time an instances is started, that is start is executed. The return value of the script does not affect the pgenv workflow. The script gets the current PGDATA path as first argument.

• PGENV_SCRIPT_PRESTOP is an executable script executed before an instance is stopped, that is before the stop command is executed. The return value of the script does not affect the pgenv workflow, and if the script generates errors they are ignored, so that the cluster is going to be stopped anyway. The script gets the current PGDATA path as first argument.

• PGENV_SCRIPT_POSTSTOP is an executable script executed each time an instance is stopped, that is the stop command is executed. The return value of the script does not affect the pgenv workflow. The script gets the current PGDATA path as first argument.

• PGENV_SCRIPT_POSTRESTART is an executable script executed each time an instance is restarted, that is the restart command is executed. The return value of the script does not affect the pgenv workflow. The script gets the current PGDATA path as first argument.

The above configuration variables can be set in the configuration file or on the fly, for instance:

$ export PGENV_SCRIPT_POSTSTART=/usr/local/bin/load_data.sh
$ pgenv start 12.0

It is worth noting that the start, restart and post script are not shared between similar events: for instance the PGENV_SCRIPT_POSTSTOP is not executed if the user issues a restart command (even if that implies somehow a cluster stop).

Please note that running external scripts can result in damages and crashes in the pgenv workflow. In the future, more hooks to run scripts could be added.

Upgrading

You can upgrade your installation to the cutting-edge version at any time with a simple git pull.

$ cd ~/.pgenv
$ git pull

Dependencies

---

• env - Sets environment for execution
• bash - Command shell interpreter
• curl - Used to download files
• sed, grep, cat, tar, sort, tr, uname, tail - General Unix command line utilities
• patch - For patching versions that need patching
• make - Builds PostgreSQL

Optional dependencies:

• Perl 5 - To build PL/Perl
• Python - To build PL/Python

Command Reference

Like git, the pgenv command delegates to subcommands based on its first argument.

Some commands require you to specify the PostgreSQL version to act on. You can specify the version the command applies to by either entering the PostgreSQL version number or by specifying any of the special keywords:

• current or version to indicate the currently selected PostgreSQL version;
• earliest to indicate the oldest installed version (excluding beta versions);
• newest to indicate the newest installed version (excluding beta versions).

It is important to note that earliest and latest have nothing to do with the time you installed PostgreSQL by means of pgenv: they refer only to PostgreSQL stable versions. To better clarify this, the following snippet shows you which aliases point to which versions in an example installation.

9.6.19                 <-- earliest (also earliest 9.6)
9.6.20      <-- latest 9.6
12.2        <-- earliest 12
12.4        <-- latest 12
13beta2
13.0                   <-- latest (also latest 13)

The subcommands are:

pgenv use

Sets the version of PostgreSQL to be used in all shells by symlinking its directory to ~/$PGENV_ROOT/pgsql and starting it. Initializes the data directory if none exists. If another version is currently active, it will be stopped before switching. If the specified version is already in use, the use command won't stop it, but will initialize its data directory and starts it if it's not already running.

$ pgenv use 10.4
waiting for server to shut down.... done
server stopped
waiting for server to start.... done
server started
PostgreSQL 10.4 started

The use command supports the special keywords earliest and latest to respectively indicate the oldest PostgreSQL version installed and the newest one. It is also possible to indicate a major version to narrow the scope of the special keywords. As an example:

pgenv use latest 10

will select the most recent PostgreSQL version of the 10 series installed.

pgenv switch

Sets the version of PostgreSQL to be used in all shells by symlinking its directory to $PGENV_ROOT/pgsql. Contrary to pgenv use this command does not manage a database for you. Meaning, it will not start, stop and initialize a postgres database with the given version. Instead it simply changes the environment to a different version of PostgreSQL. This can be useful if the user has other tools to automate the provisioning and lifecycle of a database.

$ pgenv switch 10.4

pgenv versions

Lists all PostgreSQL versions known to pgenv, and shows an asterisk next to the currently active version (if any). The first column reports versions available for use by pgenv and the second lists the subdirectory of $PGENV_ROOT in which the each version is installed:

$ pgenv versions
      10.4      pgsql-10.4
      11beta3   pgsql-11beta3
      9.5.13    pgsql-9.5.13
  *   9.6.9     pgsql-9.6.9

In this example, versions 9.5.13, 9.6.9, 10.4, and 11beta3 are available for use, and the * indicates that 9.6.10 is the currently active version. Each version is installed in a pgsql- subdirectory of $PGENV_ROOT.

pgenv current

Displays the currently active PostgreSQL version.

$ pgenv current
10.4

Please note that version is a command synonym for version:

$ pgenv version
10.4

pgenv clear

Clears the currently active version of PostgreSQL. If the current version is running, clear will stop it before clearing it.

$ pgenv clear
waiting for server to shut down.... done
server stopped
PostgreSQL stopped
PostgreSQL cleared

pgenv build

Downloads and builds the specified version of PostgreSQL and its contrib modules, as far back as version 8.0. It is possible to instrument the build process to patch the source tree, see the section on patching later on. If the version is already built, it will not be rebuilt; use clear to remove an existing version before building it again.

$ pgenv build 10.3
# [Curl, configure, and make output elided]
PostgreSQL 10.3 built

The build phase can be customized via a configuration file, in the case the system does not find a configuration file when the build is executed, a warning is shown to the user to remind she can edit a configuration file and start over the build process:

$ pgenv build 10.3
  ...
WARNING: no configuration file found for version 10.3
HINT: if you wish to customize the build process please
stop the execution within 5 seconds (CTRL-c) and run
    pgenv config write 10.3 && pgenv config edit 10.3
adjust 'configure' and 'make' options and flags and run again
    pgenv build 10.3

Within the configuration file it is possible to instrument the build phase from the configuration to the actual build. For instance, in order to build with PL/Perl, it is possible to configure the variable PGENV_CONFIGURE_OPTIONS adding --with-perl. Or say you need SSL support and to tell teh compiler to use Homebrew-installed OpenSSL. Edit it something like:

PGENV_CONFIGURE_OPTIONS=(
    --with-perl
    --with-openssl
    'CFLAGS=-I/opt/local/opt/openssl/include -I/opt/local/opt/libxml2/include'
    'LDFLAGS=-L/opt/local/opt/openssl/lib -L/opt/local/opt/libxml2/lib'
)

Please note that it is possible to pass argument variables within the command line to instrument the build phase. As an example, the following is a possible workflow to configure and build a customized 10.5 instance:

$ pgenv config write 10.5
pgenv config edit 10.5
# adjust PGENV_CONFIGURE_OPTIONS

$ pgenv build 10.5

In the case you need to specify a particular variable, such as the Perl interpreter, pass it on the command line at the time of build:

PERL=/usr/local/my-fancy-perl pgenv build 10.5

Patching

pgenv can patch the source tree before the build process starts. In particular, the patch/ folder can contain a set of index files and patch to apply. The process searches for an index file corresponding to the PostgreSQL version to build, and if found applies all the patches contained into the index.

Index files are named after the PostgreSQL version and the Operating System; in particular the name of an index file is composed as patch.<version>.<os> where version is the PostgreSQL version number (or a part of it) and os is the Operating System. As an example, patch.8.1.4.Linux represents the index used when building PostgreSQL 8.1.4 on a Linux machine. To provide more flexibility, the system searches for an index that is named after the exact PostgreSQL version and Operating System or a mix of possible combinations of those. As an example, in the case of the PostgreSQL version 8.1.4 the index files is searched among one of the following:

$PGENV_ROOT/patch/index/patch.8.1.4.Linux
$PGENV_ROOT/patch/index/patch.8.1.4
$PGENV_ROOT/patch/index/patch.8.1.Linux
$PGENV_ROOT/patch/index/patch.8.1
$PGENV_ROOT/patch/index/patch.8.Linux
$PGENV_ROOT/patch/index/patch.8

This allows you to specify an index for pretty much any combination or grouping desired. The first index file that matches wins and it is the only one used for the build process. If no index file is found at all, no patching is applied on the source tree.

The index file must contain a list of patches to apply, that is file names (either absolute or relative to the patch/ subfolder). Each individual file is applied thru patch(1).

It is possible to specify a particular index file, that means avoid the automatic index selection, by either setting the PGENV_PATCH_INDEX variable on the command line or in the configuration file. As an example

$ PGENV_PATCH_INDEX=/src/my-patch-list.txt pgenv build 10.5

Build special keywords

The build command accepts the special keywords earliest and latest as indicators of the version to build. The logic is as follows:

• latest triggers the build of the very last available PostgreSQL version available for download;
• earliest triggers the build of the very first available PostgreSQL version, that is 1.08 (and probably is not what you are looking for);
• latest xx where xx is a PostgreSQL major version number (e.g., 13) triggers the build of the latest available version in such major version branch (e.g., 13.1);
• earliest xx where xx is a PostgreSQL major version number (e.g., 13) triggers the build of the earliest available version in such major version branch (e.g., 13.0).

The latest and earliest keywords work only with the build command and not with the rebuild command.

pgenv rebuild

The rebuild command allows the rebuilding from sources of a specific PostgreSQL version. The PGDATA directory will not be deleted if already initialized via initdb. However, in the case the PostgreSQL instance to rebuild is currently in use, the rebuild command will not proceed. This is meant to prevent the user to change the binaries of a in-use PostgreSQL cluster.

The configuration will follow the same rules adopted in build, which means if a configuration file is present it will be loaded, otherwise the system will claim about such file proposing to create one.

In the case a specific version has never been built, rebuild acts exactly as build.

pgenv remove

Removes the specified version of PostgreSQL unless it is the currently-active version. Use the clear command to clear the active version before removing it.

$ pgenv remove 10.3
PostgreSQL 10.3 removed

The command removes the version, data directory, source code and configuration.

The remove command supports the special keywords earliest and latest to respectively indicate the oldest PostgreSQL version installed and the newest one. It is also possible to indicate a major version to narrow the scope of the special keywords. As an example:

pgenv remove latest 10

will remove the most recent PostgreSQL version of the 10 series installed.

pgenv start

Starts the currently active version of PostgreSQL if it's not already running. Initializes the data directory if none exists.

$ pgenv start
PostgreSQL started

It is possible to specify flags to pass to pg_ctl(1) when performing the START action, setting the PGENV_START_OPTIONS array in the configuration. Such options must not include the data directory, nor the log file.

pgenv stop

Stops the currently active version of PostgreSQL.

$ pgenv stop
PostgreSQL 10.5 stopped

It is possible to specify flags to pass to pg_ctl(1) when performing the stop action, setting the PGENV_STOP_OPTIONS array in the configuration.

pgenv restart

Restarts the currently active version of PostgreSQL, or starts it if it's not already running.

$ pgenv restart
PostgreSQL 10.1 restarted
Logging to pgsql/data/server.log

It is possible to specify flags to pass to pg_ctl(1) when performing the restart action, setting the PGENV_RESTART_OPTIONS array in the configuration.

pgenv available

Shows all the versions of PostgreSQL available to download and build. Handy to help you finding a version to pass to the build command. Note that the available command produces copious output.

$ pgenv available

            Available PostgreSQL Versions
========================================================
                  ...
 
                    PostgreSQL 9.6
    ------------------------------------------------
    9.6.0   9.6.1   9.6.2   9.6.3   9.6.4   9.6.5
    9.6.6   9.6.7   9.6.8   9.6.9   9.6.10

                    PostgreSQL 10
    ------------------------------------------------
    10.0    10.1    10.2    10.3    10.4    10.5

                    PostgreSQL 11
    ------------------------------------------------
     11.0    11.1    11.2    11.3    11.4    11.5   
     11.6    11.7    11.8    11.9    11.10   11.11  
     11.12   11.13  

                     PostgreSQL 12
    ------------------------------------------------
     12.0    12.1    12.2    12.3    12.4    12.5   
     12.6    12.7    12.8   

                     PostgreSQL 13
    ------------------------------------------------
     13.0    13.1    13.2    13.3    13.4   

                     PostgreSQL 14
    ------------------------------------------------
     14beta1  14beta2  14beta3  14rc1   14.0   

     

The versions are organized and sorted by major release number. Any listed version may be passed to the build command.

To limit the list to versions for specific major releases, pass them to available. For example, to list only the 9.6 and 10 available versions:

$ pgenv available 10 9.6
            Available PostgreSQL Versions
========================================================

                    PostgreSQL 9.6
    ------------------------------------------------
    9.6.0   9.6.1   9.6.2   9.6.3   9.6.4   9.6.5
    9.6.6   9.6.7   9.6.8   9.6.9   9.6.10

                    PostgreSQL 10
    ------------------------------------------------
    10.0    10.1    10.2    10.3    10.4    10.5

pgenv check

Checks the list of commands required to download and build PostgreSQL. Prints a result for each, with either the path to the command or an error reporting that the command was not found.

pgenv help

Outputs a brief usage statement and summary of available commands, like the following:

$ pgenv help
Usage: pgenv <command> [<args>]

The pgenv commands are:
    use        Set and start the current PostgreSQL version
    start      Start the current PostgreSQL server
    stop       Stop the current PostgreSQL server
    restart    Restart the current PostgreSQL server
    switch     Set the current PostgreSQL version
    clear      Stop and unset the current PostgreSQL version
    build      Build a specific version of PostgreSQL
    rebuild    Re-build a specific version of PostgreSQL
    remove     Remove a specific version of PostgreSQL
    version    Show the current PostgreSQL version
    current    Same as 'version'
    versions   List all PostgreSQL versions available to pgenv
    help       Show this usage statement and command summary
    available  Show which versions can be downloaded
    check      Check all program dependencies
    config     View, edit, delete the program configuration
    log        Inspects the log of the cluster, if exist.

For full documentation, see: https://github.com/theory/pgenv#readme

This is 'pgenv' version 1.0.0 [5839e72]

The last line of the 'help' shows the pgenv version number and, if git is available, the short commit hash (this can be useful when reporting bugs and filling issue requests). Please note that, in order to print out the git HEAD information, the pgenv must be able to find a git executable (i.e., it must be in your PATH) and the PGENV_ROOT must be a git checkout directory.

pgenv config

View, set, and delete configuration variables, both globally or for specific versions of PostgreSQL. Stores the configuration in Bash files, one for each version, as well as a default configuration. If pgenv cannot find a configuration variable in a version-specific configuration file, it will look in the default configuration. If it doesn't find it there, it tries to guess the appropriate values, or falls back on its own defaults.

The config command accepts the following subcommands:

• show prints the current or specified version configuration
• init produces a configuration file from scratch, with default settings
• write store the specified version configuration
• edit opens the current or specified version configuration file in your favourite text editor (Using $EDITOR, e.g: export EDITOR=/usr/bin/emacs)
• delete removes the specified configuration
• migrate is a command used to change the configuration format between versions of pgenv
• path accepts a version number and prints on standard output the path to such version configuration path

Each sub-command accepts a PostgreSQL version number (e.g., 10.5) or a special keyword:

• current or version tells pgenv to use the currently active version of PostgreSQL
• default tells pgenv to use the default configuration;
• earliest and latest to indicate respectively the oldest or newest version of PostgreSQL installed. As in other commands, these two keywords can be combined with a PostgreSQL major version number to point to the configuration of the earliest/latest version within that major number.

If no version is explicitly passed to any of the config subcommands, the program will work against the currently active version of PostgreSQL.

In order to start with a default configuration, use the write subcommand:

$ pgenv config write default
pgenv configuration file ~/.pgenv/config/default.conf written

A subsequent show displays the defaults:

$ pgenv config show default
# Default configuration
# pgenv configuration for PostgreSQL
# File: /home/luca/git/misc/PostgreSQL/pgenv/config/default.conf
# ---------------------------------------------------
# pgenv configuration created on mer 12 set 2018, 08.35.52, CEST

# Enables debug output
# PGENV_DEBUG=''

###### Build settings #####
# Make command to use for build
# PGENV_MAKE=''

# Make flags
PGENV_MAKE_OPTIONS=(-j3)

# Configure flags
# PGENV_CONFIGURE_OPTIONS=( )
# ...

##### Runtime options #####
# Path to the cluster log file
PGENV_LOG='/home/luca/git/misc/PostgreSQL/pgenv/pgsql/data/server.log'

# Cluster administrator user
PGENV_PG_USER='postgres'

# Initdb flags
PGENV_INITDB_OPTIONS=(-U postgres --locale en_US.UTF-8 --encoding UNICODE)

# ...

You can edit the file and adjust parameters to your needs.

In order to create a configuration file for a specific version, it is possible to use the init or write commands. If no prior default configuration exists, the commands do the same, that is they create a from-scratch configuration file. If a default configuration exists, the write command will "clone" such configuration in a PostgreSQL version specific configuration file, while init will create a configuration file with default settings. After a configuration file has been created by init, the write or edit commands must be used against it, that means an existing configuration file cannot be inited more than once.

$ pgenv config write 10.5
pgenv configuration file [~/.pgenv/config/10.5.conf] written

Each time pgenv writes a configuration file, it first creates a backup with the suffix .backup and a timestamp string related to when the backup file has been created.

Use the edit subcommand to edit a configuration file in your favorite editor:

pgenv config edit 10.5

The edit command will start your favorite editor, that is whatever it is set within the EDITOR variable. If such variable is not set you will be warned. Use the delete subcommand to delete a configuration:

$ pgenv config delete 10.5
Configuration file ~/.pgenv/config/10.5.conf (and backup) deleted

The delete subcommand will not attempt to delete the default configuration file, since it can be shared among different PostgreSQL versions. However, if it is explicitly specified default as the version to delete (i.e., config delete default), the default configuration file will be deleted.

$ pgenv config delete
Cannot delete default configuration while version configurations exist
To remove it anyway, delete ~/.pgenv/config/default.conf.

The delete subcommand deletes both the configuration file and its backup copy. The pgenv remove command also deletes any configuration for the removed version.

Please note that since commit 5839e721 the file name of the default configuration file has changed. In the case you want to convert your default configuration file, please issue a rename like the following

cp .pgenv.conf .pgenv.default.conf

The migrate command allows pgenv to change the configuration format of the files between different releases. For example, it must be run if you are upgrading pgenv from a version before 1.2.1 [811ba05], that changed the location of configuration files into the config subdirectory.

pgenv config migrate
Migrated 3 configuration file(s) from previous versions (0 not migrated)
Your configuration file(s) are now into [~/git/misc/PostgreSQL/pgenv/config]

The path command accepts a specific version that will be used to compute the configuration file name related to such version, printing out the resulting path to the configuration file. This allows the user to set the PGENV_CONFIGURATION_FILE environment variable to a specific path to a custom configuration file, so that other subsequent invocations of pgenv will refer to such path. As an example, this is a way to exploit the default configuration file for different versions of PostgreSQL. In order to export the variable to a specific custom file location you can do, in your terminal, something like the following:

export PGENV_CONFIGURATION_FILE=$( pgenv config path 15.4 )

The above will set the environment variable PGENV_CONFIGURATION_FILE to the configuration file for the PostgreSQL version 15.4. If you want to use the default configuration file, substitute the version number with the special default keyowrd, for example:

export PGENV_CONFIGURATION_FILE=pgenv config path default

pgenv log

The log command provides a dump of the cluster log, if it exists, so that you don't have to worry about the exact log location. The log is dumped using the tail command, and every option passed to the command line is passed thru tail. As an example:

$ pgenv log     
Dumping the content of /home/luca/git/misc/PostgreSQL/pgenv/pgsql/data/server.log 

LOG:  could not bind IPv4 address "127.0.0.1": Address already in use
HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
WARNING:  could not create listen socket for "localhost"
FATAL:  could not create any TCP/IP sockets
LOG:  database system is shut down
LOG:  starting PostgreSQL 12.1 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 8.3.0-6ubuntu1) 8.3.0, 64-bit
LOG:  listening on IPv4 address "127.0.0.1", port 5432
LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
LOG:  database system was shut down at 2020-08-28 12:57:33 CEST
LOG:  database system is ready to accept connections

The above is equivalent to manually executing

tail ~/git/misc/PostgreSQL/pgenv/pgsql/data/server.log

It is possible to pass arguments to tail as command line flags:

$ pgenv log -n 2
Dumping the content of /home/luca/git/misc/PostgreSQL/pgenv/pgsql/data/server.log 

LOG:  database system was shut down at 2020-08-28 12:57:33 CEST
LOG:  database system is ready to accept connections

which results in executing

tail -n 2 /home/luca/git/misc/PostgreSQL/pgenv/pgsql/data/server.log 

and of course, you can inspect the log of live system continuously:

pgenv log -f

Bug Reporting

Please use GitHub issues.

See Also

• plenv is a binary manager for Perl, and was the inspiration for pgenv.
• plenv, in turn, was inspired by and based on rbenv, a binary manager for Ruby.
• Pgenv works similarly, but requires PostgreSQL manually compiled from its Git repo.

License

Distributed under The MIT License; see LICENSE.md for terms.

pgenv's Issues

Configuration not correctly loaded when rebuilding postgres

When rebuilding a specific postgres version it seems that the configuration does not get correctly loaded.

Given the following config/default.conf:

PGENV_CONFIGURE_OPTIONS=(
    --enable-cassert
)

When I run pgenv build 14.2 we get a compiled version of postgres 14.2 with cassert enabled. Simplest way to confirm the build configuration I know is via pg_config

$ pgsql-14.2/bin/pg_config --configure
 '--prefix=/Users/nilsdijk/.pgenv/pgsql-14.2' '--enable-cassert' 'CFLAGS=-I/usr/local/include/ -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include/ -I/usr/local/include/ -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include/ ' 'LDFLAGS=-L/usr/local/opt/gettext/lib -L/usr/local/opt/zstd/lib -L/usr/local/opt/lz4/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/icu4c/lib -L/usr/local/opt/[email protected]/lib' 'CPPFLAGS=-I/usr/local/include -I/usr/local/opt/gettext/include -I/usr/local/opt/zstd/include -I/usr/local/opt/lz4/include -I/usr/local/opt/readline/include -I/usr/local/opt/icu4c/include -I/usr/local/opt/[email protected]/include -I/usr/local/include -I/usr/local/opt/gettext/include -I/usr/local/opt/zstd/include -I/usr/local/opt/lz4/include -I/usr/local/opt/readline/include -I/usr/local/opt/icu4c/include -I/usr/local/opt/[email protected]/include ' 'PKG_CONFIG_PATH=:/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/lz4/lib/pkgconfig:/usr/local/opt/zstd/lib/pkgconfig:/usr/local/opt/gettext/lib/pkgconfig:/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/lz4/lib/pkgconfig:/usr/local/opt/zstd/lib/pkgconfig:/usr/local/opt/gettext/lib/pkgconfig'

Ignoring the CFLAGS/LDFLAGS/CPPFLAGS that I have in my environment we can clearly see the --enable-cassert as a second configure flag being passed in.

Now we run pgenv rebuild 14.2.

Now the output of pg_config changed as follows:

$ pgsql-14.2/bin/pg_config --configure
 '--prefix=/Users/nilsdijk/.pgenv/pgsql-14.2' 'CFLAGS=-I/usr/local/include/ -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include/ -I/usr/local/include/ -I/usr/local/opt/readline/include -I/usr/local/opt/[email protected]/include/ ' 'LDFLAGS=-L/usr/local/opt/gettext/lib -L/usr/local/opt/zstd/lib -L/usr/local/opt/lz4/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/icu4c/lib -L/usr/local/opt/[email protected]/lib' 'CPPFLAGS=-I/usr/local/include -I/usr/local/opt/gettext/include -I/usr/local/opt/zstd/include -I/usr/local/opt/lz4/include -I/usr/local/opt/readline/include -I/usr/local/opt/icu4c/include -I/usr/local/opt/[email protected]/include -I/usr/local/include -I/usr/local/opt/gettext/include -I/usr/local/opt/zstd/include -I/usr/local/opt/lz4/include -I/usr/local/opt/readline/include -I/usr/local/opt/icu4c/include -I/usr/local/opt/[email protected]/include ' 'PKG_CONFIG_PATH=:/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/lz4/lib/pkgconfig:/usr/local/opt/zstd/lib/pkgconfig:/usr/local/opt/gettext/lib/pkgconfig:/usr/local/opt/icu4c/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig:/usr/local/opt/lz4/lib/pkgconfig:/usr/local/opt/zstd/lib/pkgconfig:/usr/local/opt/gettext/lib/pkgconfig'

Again ignore the CFLAGS/LDFLAGS/CPPFLAGS we find that the --enable-cassert flag is missing.

In the mean time we see two new config files created in the conf directory.

config/14.2.conf

...
# Configure flags, including PL languages but without --prefix
declare -a PGENV_CONFIGURE_OPTIONS=()
...

config/14.2.conf.2022-05-04T13-54.backup

...
# Configure flags, including PL languages but without --prefix
declare -a PGENV_CONFIGURE_OPTIONS=([0]="--enable-cassert")
...

The backup seems to be indicating that at some point in time the flag was correctly stored in the version specific configuration, but unused during the rebuild as per output of pg_config --configure. To be completely clear, the only configuration file I have created is config/default.conf, the others are (probably?) created by pgenv during builds.

I ran these experiments on OSX 12.3.1, with zsh as a shell.
Both tried bash versions:

• 5.1.16(1)-release installed via brew
• 3.2.57(1)-release that came with OSX

Quick experiments and asking around show the same behaviour on Ubuntu (docker ubuntu:22.04).

"pgenv check" needs check gcc existence too?

checking for cc... no
configure: error: in /home/marcelo/.pgenv/src/postgresql-10.5': configure: error: no acceptable C compiler found in $PATH See config.log' for more details

Split script

Now that the script is growing, should we move all the pgenv_xxx functions to a separted file to be sourced in the beginning of the script?
Advantages: the code of the script will shrink and be more readable.
Disadvantages: the code is moved across two files.

Thoughts?

pg_hba.conf & postgresql.conf

Hello... I just started using pgenv as a database server that I have various different versions of postgres built so I can change between them, and restore version specific database backups. The problem is that our backups don't contain the pg_hba.conf or the postgresql.conf so I have to make sure I have copies of those two config files in the base pgsql-## and then copy them into the data directory after I have extracted the backup.

Is there a way to tell pgenv to look for these 2 specific files in a different location that I can keep static? If not is it possible to add this as an enhancement. If there are other required configuration files for postgres to initialize correctly then probably include those as well.

Thanks in advance for any assistance!

Install postgis

It would be nice if we could install extensions like postgis with pgenv. Is there any plan to do so?

Add Support support for additional environment variables

In order to build Postgres 16, I added this environment variable to default.conf:

export PKG_CONFIG_PATH="/opt/homebrew/opt/icu4c/lib/pkgconfig"

This works fine for the build, but not any subsequent build, because the variable isn't copied to the version-specific config. I'm wondering if we could have one of two things happen (or maybe support both):

• Copy non-PGENV variables set in the default config to the version-specific one. Might require some additional PGENV_ variable to create a list of such variables?
• Prevent the creation and use of version-specific config files. I just don't need them, so might be nice to have an option for pgenv not to create them automatically, only via pgenv config write

Thoughts?

Maybe don't delete .pgenv.default.conf

I'm setting up a new computer so starting from scratch. I installed 14, then pgenv removed it, and was surprised to see that it deleted .pgenv.default.conf. It probably should not, right?

Also, pgenv write does not load any config files before it writes them. Shouldn't it at least load .pgenv.default.conf first?

use '-mf' on stop

Any reason not to use -mf when the script calls pg_ctl stop ? Or maybe allow us to pass args:

pgenv stop -mf

That way we can pass -mf or -mi or -ms as needed?

why show the 'Using PGENV_ROOT /something' msg every time?

Just curious why every single invocation of the tool shows the 'Using PGENV_ROOT /something' message? Shouldn't this be hidden behind a verbose or debug flag? It makes using pgenv in scripting a bit of a pain as you have to throw away line 1 all the time.

Tag releases?

We should start to tag version releases? In the case, what versioning schema should we adopt?

FYI for WSL -> set core.autocrlf false

This works on WSL, thank you so much; I have 11beta4 built and running.

A quick note for the WSL scenario: before cloning the repository make sure git config --global core.autocrlf false otherwise the script errors out on line endings; or at least that was my experience, and the fix as well.

Changes in PGENV_INITDB_OPTIONS (config-file) are not applied

Hello,

first of all thank you for the great tool. I am pretty new to the tool and make my first steps with it.

My environment is:

OS: Redhat Linux 8.6 x64
Kernel: 4.18.0-372.19.1.el8_6.x86_64
Shell: Bash
User: postgres
pgenv-Version: 1.3.1

In default.conf (under directory config) I changed Initdb-Flags to:
declare -a PGENV_INITDB_OPTIONS=([0]="-U" [1]="postgres" [2]="--locale" [3]="de_DE.UTF-8" [4]="--encoding" [5]="UNICODE" [6]="--data-checksums")

Unfortuntely the new parameters didn't make it to the execution. Instead the standard values in the pgenv-script are used.

Changes of initdb-parameters will be applied if I change PGENV_INITDB_OPTIONS in the pgenv-script directly.

But I think the better solution would be the expected way with the change in the config-file.

The same problem seems to occur with changes in PGENV_CONFIGURE_OPTIONS.

I tried to debug the situation a little bit and it seems to me that the changes in the config-file are visible (exported) as long as I am in the function "pgenv_configuration_load" but get lost as soon as I leave the function.

But maybe it's a mistake on my side and I'm doing something wrong.

Best regards
Kai

Installation via Homebrew

Are there any plans to support Homebrew?

No uuid-ossp extension

create EXTENSION "uuid-ossp";
ERROR:  could not open extension control file "/Users/hooopo/.pgenv/pgsql-10.1/share/extension/uuid-ossp.control": No such file or directory

it seems no contrib module installed?

my pg version 10.1；max os majave

Unset Locale settings. Is it important to check before build?

O.S: Ubuntu 18.04

make[4]: Leaving directory '/home/marcelo/.pgenv/src/postgresql-10.5/src/fe_utils'
'/usr/bin/perl' create_help.pl ../../../doc/src/sgml/ref sql_help
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_NAME = "es_AR.UTF-8",
LC_NUMERIC = "es_AR.UTF-8",
LC_TIME = "es_AR.UTF-8",
LC_MONETARY = "es_AR.UTF-8",
LC_TELEPHONE = "es_AR.UTF-8",
LC_PAPER = "es_AR.UTF-8",
LC_MEASUREMENT = "es_AR.UTF-8",
LC_IDENTIFICATION = "es_AR.UTF-8",
LC_ADDRESS = "es_AR.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").

Support multiple CFLAGS in PGENV_CONFIGURE_OPTS

I want to add multiple CFLAGS to my configure options. However I failed to find a reasonable way to place it in my .pgenv.default.conf file.

Currently I have the following line in .pgenv.default.conf

PGENV_CONFIGURE_OPTS='--enable-cassert --enable-depend --enable-debug --with-icu --with-libxml --with-libxslt --with-openssl --with-uuid=e2fs'

I also want to set CFLAGS but pgenv fails to parse my PGENV_CONFIGURE_OPTS values such as:

PGENV_CONFIGURE_OPTS='--enable-cassert --enable-depend --enable-debug --with-icu --with-libxml --with-libxslt --with-openssl --with-uuid=e2fs CFLAGS="-Og -g3 -fno-omit-frame-pointer -gdwarf"'

How do you recommend I escape the whitespaces properly?

---

My current workaround is setting the CFLAGS on bash, and lose track of those flags later as they are not stored in any files. For example:

bash$ CFLAGS="-Og -g3 -fno-omit-frame-pointer -gdwarf"  pgenv rebuild 11.10

Installing extensions such as Postgis and PGRouting

We have a product where our production environment uses Docker and has no issue with installing things like postgis and pgrouting.

However, we have multiple remote developers and many of them (including me) are using pgenv for our PostgreSQL install and I don't know how to get those extensions installed properly. The error looks like this:

acme_test=# create extension if not exists postgis;
ERROR:  could not open extension control file "/home/ovid/.pgenv/pgsql-11.5/share/extension/postgis.control": No such file or directory

I have several postgis.control files, but I don't know how to link them (or if it's safe to do so):

12:11:34 {spike-pgrouting-120} ~/client/acme-api $ locate postgis.control
/etc/alternatives/postgresql-12-postgis.control
/usr/share/postgresql/12/extension/postgis.control
/var/lib/dpkg/alternatives/postgresql-12-postgis.control

I've installed postgis via sudo apt-get install postgis.

This is Postgresql 11.5 on Ubuntu 18.04.4.

declare -g not Supported on macOS

In a547e3e, this change was made:

-        typeset -p "${name}" >> "$file"
+        # declare has no way to output a global variable
+        # that is then needed when reloading configuration !
+        declare  -p "${name}" | sed 's/declare/declare -g /' >> "$file"

However, the -g option to declare is not supported on macOS:

$ pgenv use 14.4     
/Users/david/.pgenv/config/14.4.conf: line 14: declare: -g: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Is the -g option needed for something? Maybe we should use export instead?

Multiple installations of the same database version

It would be very nice to install the same version of PostgreSQL more than once.

Workflow:

1. pgenv build-duplicate 9.6.2 (creates a 9.6.2-v$version if it's not the first one)
2. pgenv alias veure 9.6.2-v$version (#31)
3. bin/setup/install-sqitch
4. sqitch deploy
5. Run tests against the new database
6. While tests are running, go to step one to "pre-prepare" for another Jenkins run rather than waiting for the same database

This would likely require separate data directories to be really useful.

Add support for development branch?

I'm wondering about the idea of adding support for the development branch (currently postgres 12).

One issue is that you need to make this development version available. One option would be to force a "development branch" version into the versions hash after it does the lookup to the default source repo. This feels a bit ugly though especially in cases where people may have pointed to a custom source directory where they don't expect this other version to appear.

Another issue is how to build it. We could build out machinery to pull from git, which might make the most sense, since that way people could rebuild the dev version anytime they saw a new commit they were interested in. The downside, of course, is you need to add a bunch of routines to compile everything. It might be possible to download a zip file directly from GitHub, but that would still take some working out. If we want to leverage the existing build code, the easiest way would be to force dev to come from https://ftp.postgresql.org/pub/snapshot/dev/, although I think that is only built nightly, not per commit.

Or maybe this is all a bad idea? Thoughts?

Move configuration file(s) to ~/.config folder?

This is partially related to #47 and the work done in #49.
So far, the configuration files are stoed in PGENV_ROOT, while it could be easy enough to migrate them to ``/.config`, that is where a user should keep her own configuration.
Advantages:

• configuration files will be kept as part of the home folder, thus she can backup and migrate to other machines with external tools;
• separating PGENV_ROOT from its configuration should prevent accidental removal of configuration by the user;
• multiple pgenv instances can share the very same configuration.

Drawbacks:

• the user will have to manually delete PGENV_ROOT and the configuration folder/files if she does not want pgenv anymore.

I could work on this in the case it is interesting.

psql_history escape sequences

$ psql
~> select 1 ;
┌──────────┐
│ ?column? │
╞══════════╡
│        1 │
└──────────┘
(1 row)

Time: 0.752 ms
~> \q

~
$ cat ~/.psql_history
_HiStOrY_V2_
select\0401\040;
\134q

General suggestions (can break into separate tickets)

This isn't to create an issue per se, but to discuss a few things I saw yesterday that are still fresh on my mind while using pgenv for the first time. If any seem relevant, happy to create separate issues for them.

First, let me just say this is an awesome tool. Thank you for it :) I don't think an issue is required for this.

New subcommand: alias

We like to have a lot of our deployment workflow for the Tau Station MMORPG involve aliases. Thus, if we want to sqitch deploy some major databases changes, we'd love to have an alias:

pgenv build $pg_version
pgenv alias $pg_version veure
pgenv use veure
bin/setup/install-sqitch
sqitch deploy

And with that, we then safely proceed with our regular deployment and restarting our servers automatically point to the correct database. If something goes wrong ...

pgenv alias $pg_previous_version veure # removes current alias

And then we can roll back our changes and restart the servers. Players may have lost some game action, but hopefully not too much.

Of course, the remove, current, and versions commands may need to be aware of the aliases.

New subcommand: rebuild

pgenv rebuild $pg_version
pgenv rebuild $pg_version --keep-config

More or less equivalent to:

pgenv clear # if you're using the current version
pgenv build $pg_version

However, the --keep-config option would make it trivial for someone to edit the $pg_version config and, say, add --with-uuid=ossp for older PostgreSQL's and not worry about the 5 second window which has bitten me several times when reinstalling with pgenv:

        # warn if no configuration was loaded
        if [ -z "$PGENV_CONFIGURATION_FILE" ]; then
            echo "WARNING: no configuration file found for version $v"
            echo "HINT: if you wish to customize the build process please"
            echo "stop the execution within 5 seconds (CTRL-c) and run "
            echo "    pgenv config write $v && pgenv config edit $v"
            echo "adjust 'configure' and 'make' options and flags and run again"
            echo "    pgenv build $v"
            echo
            sleep 5
        fi

Alternatively, have a workflow change whereby pgenv simply halts on -z "$PGENV_CONFIGURATION_FILE" and prompts if you wish to continue (with an optional flag to pgenv build to skip this check when people want deployment automated).

Warn if current data directory?

Not sure how this is handled by PostgreSQL, but this bit me hard yesterday (particularly as I was working with an old, slow computer).

I needed to upgrade PostgreSQL from 9.3.5 to 9.6.2. pgenv was a breeze my first time around, and then I got this:

$ psql taustation
Null display is "[NULL]".
Expanded display is used automatically.
psql (9.6.15) server (9.3.5)
Type "help" for help.

I actually didn't notice this until a long sqitch deploy failed midway due to an incompatible data type. This was because I had an older pg installed. So I did brew uninstall postgresql and it removed it, but the problem remained because I had an old data directory lying around (I don't know if homebrew doesn't remove that or if it was from a previous build, but the versions were the same).

I finally manually deleted the data directory and everything built nicely. However, if there's some way to detect if PostgresSQL is going to use an existing data directory, it would be nice to know that.

Update: fixing many typos in the above because I was very sick when I wrote this. Sorry if it was confusing at times.

env not available on osx

In a recent commit, this tool's shebang line switched from a hardcoded bash path to using env to find bash. Unfortunately, env doesn't ship on osx.

Typo in GH description

There is a typo in the repo description: PostgreSQL binary man_a_ger

How do I work with the compiled code?

I was able to build for PostgreSQL 10.4 with the following command below:

sudo apt install -y libreadline-dev libperl-dev libz-dev
pgenv build 10.4

OS:

Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-33-generic x86_64)

Everything is fine. But then, the problem is how to use it

configure: error: invalid package name: perl PERL

I'm running this in WSL, so maybe a long shot.
short-of-it: At the build step I get configure: error: invalid package name: perl PERL
here is some bash output for my situation:

lsb_release -a

pt@USHCC10091:~/.pgenv$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial

pgenv check

pt@USHCC10091:~/.pgenv$ pgenv check
[OK] make:      /usr/bin/make
[OK] curl:      /usr/bin/curl
[OK] patch:     /usr/bin/patch
[OK] tar:       /bin/tar
[OK] sed:       /bin/sed
[OK] perl:      /usr/bin/perl
[OK] python:    /usr/bin/python

git clone contents

pt@USHCC10091:~/.pgenv$ ls -lsa
total 24
 0 drwxrwxrwx 1 pt pt  4096 Sep 20 15:53 .
 0 drwxr-xr-x 1 pt pt  4096 Sep 20 15:56 ..
 0 drwxrwxrwx 1 pt pt  4096 Sep 20 15:27 bin
 0 drwxrwxrwx 1 pt pt  4096 Sep 20 15:27 .git
 0 -rw-rw-rw- 1 pt pt    23 Sep 20 15:27 .gitignore
 4 -rw-rw-rw- 1 pt pt  1081 Sep 20 15:27 LICENSE.md
 4 -rw-rw-rw- 1 pt pt  1180 Sep 20 15:53 .pgenv.conf
16 -rw-rw-rw- 1 pt pt 14809 Sep 20 15:27 README.md
 0 drwxrwxrwx 1 pt pt  4096 Sep 20 15:54 src

do the build

pt@USHCC10091:~/.pgenv$ pgenv build 10.5
configure: error: invalid package name: perl  PERL

check perl version

pt@USHCC10091:~/.pgenv$ perl -v

This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi
(with 69 registered patches, see perl -V for more detail)

Copyright 1987-2015, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using "man perl" or "perldoc perl".  If you have access to the
Internet, point your browser at http://www.perl.org/, the Perl Home Page.

Add patches for macOS?

Look at the changes required for PL/Perl and PL/TCL on macOS 10.14 “Mojave”. Pretty annoying. The next releases of currently-supported versions of Postgres will include this fix, but not older versions, of course. Do we need to create patches for those versions?

To that end, do we need to formalize a patching process? Right now we just patch 8.0 and 8.1, but it's feasible we'll need to create and maintain patches for other versions. Perhaps we should create some formal process. Maybe something like creating a patch directory structure? That's fine for targeting specific versions, or major versions, but what about a case like this, where a fix will be in a core release soon, and we won't need to apply the patch for those newer versions? Maybe create some sort of mapping file?

Instrument the creation of the configuration file

pgenv creates a new per-PostrgeSQL configuration file as soon as it builds/rebuilds the version.
When a PostgreSQL version is used/started, pgenv looks for the per-version configuration file and if found loads it, or it loads the file as pointed out from the environment variable PGENV_CONFIGURATION_FILE (see #61).
It could be useful to prevent at all pgenv form creating per-version configuration files, for example if the user wants to use always a specific configuration file and does not care about having unused files laying around (see discussion #61 (comment)).

The behavior can be achieved in different ways:

• use an environment variable, for example PGENV_AUTOWRITE_CONFIGURATION_FILE set to yes or no to instrument the application;
• use separate subcommand, for example build and build-no-config to distinguish about the behavior
• use some sort of command line option, lie for instance pgenv build --no-config, that will require to be able to correctly parse the command line, for example with getopt.

The first approach is the simplest one, while the last is the more extensible and can give us room to implement other options and improve commands, that so far accepts only positional arguments. For example, the latter approach could also allow the user to specify the configuration file to use on a per-command basis, like pgenv build 16beta --with-config /path/to/config instead of using the PGENV_CONFIGURATION_FILE variable.

I am towards the getopt like approach, also to avoid adding other environment variables. Clearly, this approach requires a quite extensive command line refactoring.