aiven / aiven-extras

Aiven PostgreSQL® extras

aiven-extras

This extension is meant for use in enviroments where you want non-superusers to be able to use certain database features, such as creating logical replication connections using PostgreSQL® 10+'s logical replications SUBSCRIPTION/PUBLICATION concepts or configuring the auto_explain extension.

It allows you to use logical replication when installed originally by root and the user using it has the REPLICATION privilege.

Please note that when loading the auto_explain extension, it is loaded for the current session only.

Installation

To create the Aiven extras extension, run the following after connecting to the database you wish to enable it in:

$ CREATE EXTENSION aiven_extras;
CREATE EXTENSION

Usage

The functions included are:

Name	Result data type	Argument data types
auto_explain_load	void
claim_public_schema_ownership	void
dblink_record_execute	SETOF record	text, text
dblink_slot_create_or_drop	void	arg_connection_string text, arg_slot_name text, arg_action text
explain_statement	SETOF json	arg_query text
pg_alter_subscription_disable	void	arg_subscription_name text
pg_alter_subscription_enable	void	arg_subscription_name text
pg_alter_subscription_refresh_publication	void	arg_subscription_name text, arg_copy_data boolean DEFAULT true
pg_create_publication	void	arg_publication_name text, arg_publish text, VARIADIC arg_tables text[] DEFAULT ARRAY[]::text[]
pg_create_publication_for_all_tables	void	arg_publication_name text, arg_publish text
pg_create_subscription	void	arg_subscription_name text, arg_connection_string text, arg_publication_name text, arg_slot_name text, arg_slot_create boolean DEFAULT false, arg_copy_data boolean DEFAULT true
pg_drop_subscription	void	arg_subscription_name text
pg_list_all_subscriptions	SETOF aiven_extras.aiven_pg_subscription
pg_stat_replication_list	SETOF aiven_extras.aiven_pg_stat_replication
session_replication_role	text	arg_parameter text
set_auto_explain_log_analyze	text	arg_parameter text
set_auto_explain_log_buffers	text	arg_parameter text
set_auto_explain_log_format	text	arg_parameter text
set_auto_explain_log_min_duration	text	arg_parameter text
set_auto_explain_log_nested_statements	text	arg_parameter text
set_auto_explain_log_timing	text	arg_parameter text
set_auto_explain_log_verbose	text	arg_parameter text
set_pgaudit_parameter	void	arg_parameter text, arg_database text, arg_value text
set_pgaudit_role_parameter	void	arg_parameter text, arg_role text, arg_value text

Examples

Managing subscriptions:

-- Create a new subscription
SELECT * FROM aiven_extras.pg_create_subscription(
    'subscription',
    'dbname=defaultdb host=destination-demoprj.aivencloud.com port=26882 sslmode=require user=avnadmin password=secret',
    'pub1',
    'slot',
    TRUE,
    TRUE
);

-- List all subscriptions
SELECT * FROM aiven_extras.pg_list_all_subscriptions();

-- Drop a subscription
SELECT * FROM aiven_extras.pg_drop_subscription('subscription');

-- Create a publication `pub1` with `publish='INSERT,UPDATE'` for tables `foo` and `bar`
-- (Note: use fully qualified names)
SELECT * FROM aiven_extras.pg_create_publication('pub1', 'INSERT,UPDATE', 'public.foo', 'public.bar');

-- Create a publication `pub2` with `publish='INSERT'` for all tables
SELECT * FROM aiven_extras.pg_create_publication_for_all_tables('pub2', 'INSERT');

-- Disable a subscription
SELECT * FROM aiven_extras.pg_alter_subscription_disable('subscription');

-- Enable a subscription
SELECT * FROM aiven_extras.pg_alter_subscription_enable('subscription');

-- Refresh a subscribed publication with `copy_data` set to FALSE
SELECT * FROM aiven_extras.pg_alter_subscription_refresh_publication('subscription', FALSE);

Configuring auto-explain:

For details, refer to PostgreSQL's documentation, but note that arguments for the exposed functions are of type text. Also note that currently aiven_extras implements a subset of the available functions.

-- Load the extension for the current session
SELECT * FROM aiven_extras.auto_explain_load();

-- Set minimum duration to 2000 ms
SELECT * FROM aiven_extras.set_auto_explain_log_min_duration('2000');

-- Enable ANALYZE
SELECT * FROM aiven_extras.set_auto_explain_log_analyze('on');

License

Aiven extras Extension is licensed under the Apache License, Version 2.0. Full license text is available in the LICENSE file and at http://www.apache.org/licenses/LICENSE-2.0.txt

Credits

Aiven extras extension was created by Hannu Valtonen [email protected] for Aiven Cloud Database and is now maintained by Aiven developers.

Aiven Cloud Database: https://aiven.io/

Recent contributors are listed on the GitHub project page, https://github.com/aiven/aiven-extras/graphs/contributors

Contact

Bug reports and patches are very welcome, please post them as GitHub issues and pull requests at https://github.com/aiven/aiven-extras . Any possible vulnerabilities or other serious issues should be reported directly to the maintainers [email protected].

Trademarks

Postgres, PostgreSQL and the Slonik Logo are trademarks or registered trademarks of the PostgreSQL Community Association of Canada, and used with their permission.

Copyright

Copyright (C) 2018 Aiven Ltd

aiven-extras's Issues

Not able to disable/drop subscriptions created by aiven-db-migrate

Hi there,

I was about to post this on the aiven-db-migrate project, but probably makes more sense here, I'm not sure:

Description of the issue

I am trying to reset a logical replication process from the target DB (Aiven) using aiven_extras and avnadminuser, but when I use the name (SELECT subname FROM aiven_extras.pg_list_all_subscriptions();) of the subscription in the aiven_extras.pg_drop_subscription() function, I get (redacted):

ERROR:  subscription "<subscription_name>" does not exist
CONTEXT:  SQL statement "ALTER SUBSCRIPTION <subscription_name> DISABLE"
PL/pgSQL function pg_drop_subscription(text,boolean) line 13 at EXECUTE 

SQL state: 42704

When trying to manually disable the subscription I get same error:

ERROR:  subscription "<subscription_name>" does not exist 

SQL state: 42704

And when I try to query the subscriptions, I get:

ERROR:  permission denied for table pg_subscription 

SQL state: 42501

Issue-Type (put a x sign in the square brackets)

• bug report
• feature request
• Documentation improvement
• Other

Checklist

• Running latest version of code.
• This issue has not been reported earlier.

Your environment

• OS: Alpine 3.19
• Python Version: 3.12.2
• Release tag/commit of the code: 0.1.3 (aiven-db-migrate)

Expected behaviour

the subscription should get deleted along with the publications and slots from the source database

Actual behaviour

the command fails

Steps to reproduce

Initiate a logical replication process and try to delete the subscription afterwards.

Any extra info ( for eg. code snippet to reproduce, logs, screenshots etc. )

In my case source DB is an self-managed Postgres with sudo access running on a self-managed Kubernetes cluster, target DB is an Aiven DB (Postgresql 15.5 on both) which is accessed by the aiven-db-migrate tool from an AKS cluster using VPC peering.

Not working

Hello. I'm trying to get data without using super user. I stumbled upon your solution, added your extension created a publication as indicated in the instructions, but after running I get the error io.debezium.jdbc.JdbcConnectionException: ERROR: permission denied for database project

name=debezium-test-01
tasks.max=1
plugin.name=pgoutput
publication.name=dbz_test
connector.class=io.debezium.connector.postgresql.PostgresConnector
database.hostname=10.10.10.172
database.port=5432
database.user=debezium
database.password=debezium
database.dbname=project
database.server.name=test
table.whitelist=public.test
heartbeat.interval.ms=5000
session.timeout.ms=120000
publication.autocreate.mode=filtered
table.include.list=public.test

All read and access to the use of this database is granted to the user

Any plans to publish on https://pgxn.org/?

It would be easier to manage this extension locally if it was published on https://pgxn.org/. Would be great to have a local environment similar to one of on production/staging that we get with Aiven.

Installation of aiven-extras feature to Postgres running in docker container [help wanted]

Hello,
I am asking for help with the installation of aiven-extras extension to database which is running in docker container. I am trying to dump some data from the staging database (running on digital ocean) to my local db which is running using Postgres docker image
Apparently, aiven-extras extension is installed on the staging database (which is running on digital ocean) and this extension is not a part of Postgres docker image.
Error thrown during docker build:

2023-08-01 05:24:14.374 UTC [51] ERROR:  extension "aiven_extras" is not available
2023-08-01 05:24:14.374 UTC [51] DETAIL:  Could not open extension control file "/usr/local/share/postgresql/extension/aiven_extras.control": No such file or directory.
2023-08-01 05:24:14.374 UTC [51] HINT:  The extension must first be installed on the system where PostgreSQL is running.

This error is triggered by one of these lines in schema.sql file

--
-- PostgreSQL database dump
--

-- Dumped from database version 15.3
-- Dumped by pg_dump version 15.3

-- Started on 2023-07-31 19:43:44 UTC

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SELECT pg_catalog.set_config('search_path', '', false);
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

--
-- TOC entry 6 (class 2615 OID 16985)
-- Name: aiven_extras; Type: SCHEMA; Schema: -; Owner: postgres
--

CREATE SCHEMA aiven_extras;


ALTER SCHEMA aiven_extras OWNER TO postgres;

--
-- TOC entry 2 (class 3079 OID 16986)
-- Name: aiven_extras; Type: EXTENSION; Schema: -; Owner: -
--

CREATE EXTENSION IF NOT EXISTS aiven_extras WITH SCHEMA aiven_extras;


--
-- TOC entry 4611 (class 0 OID 0)
-- Dependencies: 2
-- Name: EXTENSION aiven_extras; Type: COMMENT; Schema: -; Owner: 
--

COMMENT ON EXTENSION aiven_extras IS 'aiven_extras';

PS: I generated the schema.sql file by this line in my Dockerfile

RUN PGPASSWORD=${PROD_DB_PASSWORD} pg_dump --verbose --no-privileges -U doadmin -h ****.b.db.ondigitalocean.com -p 25060 -s core > /docker-entrypoint-initdb.d/schema.sql

auto_explain documentation?

Hello,

Could you add some documentation around using auto_explain with Aiven's Postgres service? Searching help.aiven.io I only found https://help.aiven.io/en/articles/489557-postgresql-superuser-access which links here, and while this extension does seem to define some helper functions, they are not documented as far as I can tell.

Make aiven_extras.pg_alter_subscription_refresh_publication work on pg14

We currently support ALTER SUBSCRIPTION ... REFRESH... (source) via a secutiry definer function. Unfortunately, pg14 changed this to require that this is run outside a transaction block: postgres/postgres@ce0fdbf#diff-d729c809ce03180c391eaba8a%5B%E2%80%A6%5Da292ad4daa2d06f16ce43ba1deR985 This check also prevents it to run in a function: https://github.com/postgres/postgres/blob/7b5d4c29ed0262e537026cb3a85161d6cf98abcc/src/backend/access/transam/xact.c#L3389-L3425
So If we still want to support our user to be able to run this on their own, we have to find a way to run this.

Possible options:

1. Allow pg_cron to superuser queries and use that (by submitting a cron job with SECURITY DEFINER) -> would make this functionality depend on pg_cron 😞 and leave a potential security problem in pg_cron (if someone can submit a job and get it to be run as superuser...) 😞
2. Go the pgextwlist way and hook into the SQL execution and intercept the refresh call and allow it for certain users and then execute it with superuser privs -> means changing the aiven-extra package to do C code; given the nature of this, also needs some security review and stuff...
3. Patch pg itself to ignore this check if it is run as superuser (given that only we run as superuser, we can assume/make sure that we would never run it in a transaction...)
4. (a variant of 3+4) Write our own c level function which more or less does the same as ALTER SUBSCRIPTION ... REFRESH just without the check for the function -> we basically take care to do run this outside a transaction and the code takes care to not run more stuff in the function. Needs aiven-extras to support c code...
5. USe dblink to connect to ourself (via the cert) and run the command that way.

Subscription drop with publication server down or none existing

Not really sure if this is an issue or not. But I had created publication on our old DB in AWS RDS, created a subscription on a new Aiven DB, made the switch to Aiven, then terminated the RDS before dropping the subscription on Aiven.

Now the subscription fails because the old server doesn't exists and I can't drop the subscription because the Aiven function can't drop the slot on the old server.

So on executing SELECT * FROM aiven_extras.pg_drop_subscription('subscription');

ERROR:  could not establish connection
DETAIL:  could not translate host name "data.cps7haw5lzpy.eu-central-1.rds.amazonaws.com" to address: Name or service not known
CONTEXT:  SQL statement "SELECT res FROM public.dblink('host=data.cps7haw5lzpy.eu-central-1.rds.amazonaws.com password=xxx port=5432 dbname=marketing user=xxx', 'SELECT TRUE FROM pg_catalog.pg_replication_slots WHERE slot_name = ''slot''') AS d(res BOOLEAN)"
PL/pgSQL function aiven_extras.dblink_slot_create_or_drop(text,text,text) line 14 at EXECUTE
SQL statement "SELECT aiven_extras.dblink_slot_create_or_drop(l_subconninfo, l_slot_name, 'drop')"
PL/pgSQL function aiven_extras.pg_drop_subscription(text) line 14 at PERFORM
SQL state: 08001

Anything I can do to drop the subscription while the old server is down?