gmr / pgsql-listen-exchange

RabbitMQ Exchange that publishes messages received from PostgreSQL Notifications.

PostgreSQL LISTEN Exchange

A RabbitMQ exchange type that translates PostgreSQL NOTIFY messages to AMQP messages and publishes them to bound queues. The PostgreSQL NOTIFY message channel is used as the routing key for the message using direct exchange style routing mechanics.

Example

To publish Postgres notifications as AMQP messages into queues bound to a x-pgsql-listen exchange with a binding key of test, run the following command in psql:

postgres=# NOTIFY test, 'This is a test';

Installation

Extract the contents of the release zip file into your RabbitMQ plugins directory. Once extracted, run rabbitmq-plugins enable pgsql_listen_exchange.

Configuration

Argument Based Configuration

To connect to PostgreSQL using something other than the default pgsql://postgres@localhost:5432/postgres connection, you can add arguments when declaring the exchange:

Setting	Description	Data Type
x-host	The PostgreSQL server hostname	String
x-port	The port to connect on	Number
x-dbname	The database name to connect to	String
x-user	The user to connect as	String
x-password	The password to use when connecting	String

Policy Based Configuration

To apply configuration via a policy, the following settings are available:

Setting	Description	Data Type
pgsql-listen-host	The PostgreSQL server hostname	String
pgsql-listen-port	The port to connect on	Number
pgsql-listen-dbname	The database name to connect to	String
pgsql-listen-user	The user to connect as	String
pgsql-listen-password	The password to use when connecting	String

Configuration in rabbitmq.config

You can also change the default connection values in the rabbitmq.config file:

Setting	Description	Data Type	Default Value
host	The PostgreSQL server hostname	list	"localhost"
port	The port to connect on	integer	5432
dbname	The database name to connect to	list	"postgres"
user	The user to connect as	list	"postgres"
password	The password to use when connecting	list	""

Example rabbitmq.config

[{pgsql_listen_exchange,
  [
    {host, "localhost"},
    {port, 5432},
    {dbname, "postgres"},
    {user, "postgres"},
    {password, ""}
  ]}
].

Message Properties

The exchange will automatically add the following properties to messages:

Property	Value
app_id	pgsql-listen-exchange
headers	See "Headers Properties Values" table below
timestamp	The UNIX epoch timestamp of the publishing server

Headers Property Values

The following table details the values of the headers property that is set on each message.

Key	Value
pgsql-channel	The PostgreSQL notification channel
pgsql-server	The host and port of the PostgreSQL server
source-exchange	The pgsql-listen-exchange that the notification was received by

Specifying Other Properties

In addition to the automatically set message properties, the exchange can set configured message properties. To set one of the supported message properties, specify the property name and value when binding to the exchange. For example, to set the content_type property, specify content_type and the value it should be set to when binding a queue to the exchange. The following message properties are supported:

Property	Data Type
content_encoding	String
content_type	String
delivery_mode	Number
priority	Number
reply_to	String
type	String

Building

Steps to custom build a version of the pgsql-listen-exchange plugin follow the development environment instructions at https://www.rabbitmq.com/build-server.html#prerequisites and then run the following:

make tests
make dist
zip -r pgsql-listen-exchange.zip plugins/epgsql-* plugins/pgsql-listen-exchange-*

Unzip that file into the plugins directory for RabbitMQ and enable as you would any other plugin.

pgsql-listen-exchange's Issues

Add an exchange with Invalid host name will crash the process

Rabbitmq version: 3.8.12
Plugin version: 1.0.0a1

Here is the log when add a new exchange

Request to node rabbit@rabbitmq-node-1 failed with {'EXIT',
{noproc,
{gen_server,call,
[pgsql_listen_worker,
{validate,
{exchange,
{resource,<<"/">>,
exchange,<<"x">>},
'x-pgsql-listen',
true,false,false,[],
undefined,undefined,
undefined,
{[],[]},
#{user =>
<<"guest">>}}}]}}}

the log from the node container

rabbitmq-node-2_1  | 2021-03-23 03:53:28.540 [info] <0.1431.0> supervisor: {local,pgsql_listen_sup}, errorContext: child_terminated, reason: shutdown, offender: [{pid,<0.1433.0>},{id,mirroring},{mfargs,{mirrored_supervisor,start_internal,[pgsql_listen_sup,fun rabbit_misc:execute_mnesia_transaction/1,[{pgsql_listen_worker,{pgsql_listen_worker,start_link,[]},permanent,5000,worker,[pgsql_listen_worker]}]]}},{restart_type,permanent},{shutdown,4294967295},{child_type,worker}]
rabbitmq-node-2_1  | 2021-03-23 03:53:28.542 [error] <0.1431.0> Supervisor pgsql_listen_sup had child mirroring started with mirrored_supervisor:start_internal(pgsql_listen_sup, fun rabbit_misc:execute_mnesia_transaction/1, [{pgsql_listen_worker,{pgsql_listen_worker,start_link,[]},permanent,5000,worker,[pgsql_listen_worker]}]) at <0.1433.0> exit with reason shutdown in context child_terminated
rabbitmq-node-2_1  | 2021-03-23 03:53:28.542 [info] <0.1431.0> supervisor: {local,pgsql_listen_sup}, errorContext: shutdown, reason: reached_max_restart_intensity, offender: [{pid,<0.1433.0>},{id,mirroring},{mfargs,{mirrored_supervisor,start_internal,[pgsql_listen_sup,fun rabbit_misc:execute_mnesia_transaction/1,[{pgsql_listen_worker,{pgsql_listen_worker,start_link,[]},permanent,5000,worker,[pgsql_listen_worker]}]]}},{restart_type,permanent},{shutdown,4294967295},{child_type,worker}]
rabbitmq-node-2_1  | 2021-03-23 03:53:28.543 [error] <0.1431.0> Supervisor pgsql_listen_sup had child mirroring started with mirrored_supervisor:start_internal(pgsql_listen_sup, fun rabbit_misc:execute_mnesia_transaction/1, [{pgsql_listen_worker,{pgsql_listen_worker,start_link,[]},permanent,5000,worker,[pgsql_listen_worker]}]) at <0.1433.0> exit with reason reached_max_restart_intensity in context shutdown

Seems I added the exchange with an invalid hostname with more than 5 times will crash the process and will not back to normal except restart the server

seems related to epgsql/epgsql#232

Thanks

Error on enable plugin

Hello. When trying to enable the plugin these errors occurred
{:plugins_not_found, [:pgsql_listen_exchange]}.
I'm using rabbitmq: management 3. docker. I copied the unzipped files to / opt / rabbitmq / plugins and the command rabbitmq-plugins enable pgsql_listen_exchange fails.

Could anyone help?

Incorrect use of `mirrored_supervisor`

In src/pgsql_listen_sup.erl, the following code cannot work:

stop() ->
    ok = mirrored_supervisor:terminate_child(?MODULE),
    ok = mirrored_supervisor:delete_child(?MODULE).

mirrored_supervisor:terminate_child/1 and mirrored_supervisor:delete_child/1 do not exist: they take 2 arguments.

plugin is not reconnect after postgres reboot

After reboot PostgreSQL plugin does not try to re-connect

RabbitMQ: 3.8.14
Erlang: 23.3.1
Plugin version: 1.0.0a1
PostgreSQL: 13.2

Log from rabbitmq:

2021-04-04 00:03:30.512 [info] <0.2249.1> Closing all channels from connection '<[email protected]>' because it has been closed
2021-04-04 00:06:56.002 [error] <0.27907.0> Supervisor {<0.27907.0>,mirrored_supervisor_sups} had child pgsql_listen_worker started with pgsql_listen_worker:start_link() at <0.280.1> exit with reason {shutdown,{error,fatal,<<"57P01">>,admin_shutdown,<<"terminating connection due to administrator command">>,[{file,<<"postgres.c">>},{line,<<"3090">>},{routine,<<"ProcessInterrupts">>},{severity,<<"FATAL">>}]}} in context child_terminated
2021-04-04 00:06:56.401 [error] <0.2391.1> pgsql_listen_lib:ensure_pgsql_connection/2 error: {error,fatal,<<"57P03">>,cannot_connect_now,<<"the database system is shutting down">>,[{file,<<"postmaster.c">>},{line,<<"2324">>},{routine,<<"ProcessStartupPacket">>},{severity,<<"FATAL">>}]}
2021-04-04 00:06:56.401 [error] <0.2391.1> pgsql_listen_exchange startup error for {resource,<<"/">>,exchange,<<"notification">>}: {error,fatal,<<"57P03">>,cannot_connect_now,<<"the database system is shutting down">>,[{file,<<"postmaster.c">>},{line,<<"2324">>},{routine,<<"ProcessStartupPacket">>},{severity,<<"FATAL">>}]}

Plugin fails when PostgreSQL connection is closed

Need to catch the process crashing for the pgsql connection which requires a refactor of how postgresql connections are managed.

rabbitmq.config is set but not used?

Windows environment with RabbitMQ 3.5.0, Erlang 17.4
Trying to use the rabbitmq.config file to set the db connection values. Rabbitmq shows that the config file is found, but when I try to create an exchange, an econnrefused error occurs. If I set the same values in the args of the exchange or in a policy, the exchange is created and messages appear when notify is used in the database.

Add ability to set client timezone

Out of the discussion in #26, I think it makes sense to add a client timezone setting for the Postgres connection, to force timestamps to come back in the appropriate timezone. Note that normal postgres behavior for TIMESTAMP WITHOUT TIME ZONE columns would apply.

See https://www.postgresql.org/docs/current/sql-set.html

Question regarding dropped connections

One of the issues I've had with external clients for postgres notifies is that I'd have to write code to check for dropped connections to the pgsql server and logic to discover any missed messages. So a couple questions as I'm not very familiar with either Erlang or RabbitMQ:

1. Will this plugin automatically reconnect if the connection is dropped?
2. Is it possible to pass keepalive parameters in the psql connection request?

Thank you.

restart postgres connection

I'm having an issue when I add/change the routing key used for the pgsql-listen-exchange, new messages are not being routed unless I restart RabbitMQ.

I have configure the exchange to make the connection.

Is there a way to restart the connection without having to restart RabbitMQ?

Time in 12 hour format it seems.

Hi Gavin,

I've been fighting with this and started writing code to change the timezone of the timestamp fields, however, I've just found out that I'm not going crazy.. or no more so...

What happens is the the times in the payload are correct up to 12:59. They then roll over to 1:00 instead of 13:00.

Its currently 5:10 and the current data is

{"settlementdate":"2021-03-03 05:20:00","runno":1,""lastchanged":"2021-03-03 05:15:10"} which would be totally correct if it had a pm on it???

Any idea where the heck this is happening?

Peter.

Policy based configuration bypassed on Debian Jessie

Hello.

I am having troubles using a specific configuration for pgsql-listen-exchange other
than the one provided by default with the plugin.

Despite the following configuration defined under /etc/rabbitmq/rabbitmq.config:

[{pgsql_listen_exchange,
  [
    {host, "localhost"},
    {port, 5432},
    {dbname, "my_db"},
    {user, "mu_user"},
    {password, "my_pwd"}
  ]}
].

it looks like the plugin is always using `user: postgres' without any password.
From the logs of my RabbitMQ server it looks like that the configuration is correctly
acquired:

=INFO REPORT==== 4-Sep-2015::16:44:37 ===
node           : rabbit@pgsqlsrv
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
[...]

=INFO REPORT==== 4-Sep-2015::16:46:53 ===
Server startup complete; 8 plugins started.
 * amqp_client
 * epgsql
 * mochiweb
 * pgsql_listen_exchange
[...]

But whenever I have tried to define a RabbitMQ exchange, like this:

rabbitmqadmin declare exchange name=testpgsql type=x-pgsql-listen

I will get the following error:

*** 506 pgsql_listen_worker failure (invalid_password)

And indeed, from the Postgres log file, I can see:

2015-09-05 23:19:23 CEST [31316-1] postgres@postgres FATAL:  password authentication failed for user "postgres"
2015-09-05 23:19:23 CEST [31316-2] postgres@postgres DETAIL:  User "postgres" has no password assigned.
        Connection matched pg_hba.conf line 92: "host    all             all             127.0.0.1/32  

I have also tried to change the configuration file shipped with the plugin (include/pgsql_listen.hrl)
but the result is always the same.

I am using Erlang/OTP 17.3, RabbitMQ 3.3.5 and Postgresql 9.4 (all came packaged with Debian 8).

Thanks in advance!

Add option to change delivery_mode of message

Hi,
it would be nice if we have some option which will change delivery_mode of published message to queue. Now all messages are published with delivery_mode = 1 and sometimes I need messages with delivery_mode = 2.

Error: {plugin_built_with_incompatible_erlang,"epgsql"}

As I am using RabbitMq V3.5.1.
I tried to install rabbitmq-plugins pgsql_listen_exchange which shows me following error

The following plugins have been enabled:
epgsql
pgsql_listen_exchange

Applying plugin configuration to rabbit@rpawar01-VirtualBox... failed.
Error: {plugin_built_with_incompatible_erlang,"epgsql"}

For installing plugins I followed these steps:

1. Downloaded the pgsql-listen-exchange-0.3.0-v3.5.x.zip file and extract it and I found two files i.e.
   epgsql-3.1.0-rmq0.3.0-v3.5.x-git9f52f19.ez
   pgsql_listen_exchange-0.3.0-v3.5.x.ez

2. I Placed this file in rabbitmq plugins folder i.e.
   /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.1/plugins

3. After that I have executed following command .

rabbitmq-plugins enable pgsql_listen_exchange

Which shows above error.

Please let me know if anyone has a solution for it.

How to connect PostgreSQL and RabbitMQ

Hello guys,

PostgreSQL 9.6.1
RabbitMQ 3.6.6

I am new with rabbitmq and need your help. I am trying connect my rabbitMQ container and postgreSQL container.

Could you please explain how should I configure my rabbitMQ container if PostgreSQL container configured as:

host: postgresql
port: 5432
dbname: postgres
dbuser: postgres
dbpassword: 123456789

I've tried add the

{pgsql_listen_exchange,
[
    {host, "postgresql"},
    {port, 5432},
    {dbname, "postgres"},
    {user, "postgres"},
    {password, "123456789"}
]}

to my rabbitmq.config file in the rabbitMQ container and sent from the PostgreSQL container

NOTIFY test, 'Test message';

If I understand right than my listener should listen exchange named as x-pgsql-listen with keybind 'test' in any case here is my listener code:

<?php
require_once __DIR__ . '/../../../vendor/autoload.php';
use PhpAmqpLib\Connection\AMQPStreamConnection;
$connection = new AMQPStreamConnection('rabbitmq', 5432, 'admin', '123456789');
$channel = $connection->channel();
$channel->exchange_declare('x-pgsql-listen', 'topic', false, false, false);
list($queue_name, ,) = $channel->queue_declare("something", false, false, true, false);
$binding_keys = array_slice($argv, 1);
$channel->queue_bind($queue_name, 'x-pgsql-listen', 'test');
echo ' [*] Waiting for logs. To exit press CTRL+C', "\n";
$callback = function($msg){
  echo ' [x] ',$msg->delivery_info['routing_key'], ':', $msg->body, "\n";
};
$channel->basic_consume($queue_name, '', false, true, false, false, $callback);
while(count($channel->callbacks)) {
    $channel->wait();
}
$channel->close();
$connection->close();
?>

But nothing got in console log.

The question is how should I configure my RabbitMQ container according to my PostgreSQL container configurations via configuration file or via policy configurations?

Handling of timestamps.

Hi there. I'm wondering if you can tell me what happens to timestamps as far as timezones go.

It seems that the data coming into my database as timestamp - no timezone, is hitting the rabbit queue as timestamp in UTC.

Is that correct?

Can that be changed, or do I need to assume this and handle the TZ change when I pull the data from the queue?

Thanks heaps.

Peter.

Worker Supervisor Behavior is wrong

Fix the worker process supervisor behavior to properly work with queues and not start up on nodes if it's already running in the cluster

typo in README

"Queues should be bound to the queue with the channel name as well."

you probably mean bound to the exchange

Nothing is published

I've set up the exchange and connected it to a queue.

When doing:

NOTIFY test, 'This is a test';

There are simply no messages passed into the queue.

Environment Details
rabbitmq version: 3.5.7
postgres version: 9.3.9
os: ubuntu 14.04

:(

Configuration Confusion

Not sure if this is an issue but I'm confused at to what actually needs to be configured for a connection.

Example:

Can I just configure the Policy or do I need to setup the exchange connection method as well?

Q: Do I need or Can I have more than one connection configured?

If yes, how and benefits?

Unable to setup exchange: {error, {exit, {{{econnrefused ....

RabbitMQ 3.6.1
PostgreSQL 9.5.2

rabbitmq.config

[{rabbit, [{loopback_users, []}]},
{pgsql_listen_exchange,
[
    {host, "db"},
    {port, 5432},
    {dbname, "shop"},
    {user, "postgres"},
    {password, "..."}
]}
].

I'm using web management to add the exchange name = hesting, type = x-pgsql-listen and the error I'm getting is

=ERROR REPORT==== 20-Apr-2016::09:54:34 ===
webmachine error: path="/api/exchanges/%2F/hesting"
{error,
    {exit,
        {{{econnrefused,
              {gen_server,call,
                  [pgsql_listen_worker,
                   {validate,
                       {exchange,
                           {resource,<<"/">>,exchange,<<"hesting">>},
                           'x-pgsql-listen',false,true,true,[],undefined,
                           undefined,
                           {[],[]}}}]}},
          [{gen_server,call,2,[{file,"gen_server.erl"},{line,204}]},
           {pgsql_listen_exchange,gen_server_call,1,
               [{file,"src/pgsql_listen_exchange.erl"},{line,94}]},
           {rabbit_exchange,declare,6,
               [{file,"src/rabbit_exchange.erl"},{line,168}]},
           {rabbit_channel,handle_method,3,
               [{file,"src/rabbit_channel.erl"},{line,1219}]},
           {rabbit_channel,handle_cast,2,
               [{file,"src/rabbit_channel.erl"},{line,455}]},
           {gen_server2,handle_msg,2,
               [{file,"src/gen_server2.erl"},{line,1049}]},
           {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]},
         {gen_server,call,
             [<0.2366.0>,
              {call,
                  {'exchange.declare',0,<<"hesting">>,<<"x-pgsql-listen">>,
                      false,false,true,true,false,[]},
                  none,<0.2170.0>},
              infinity]}},
        [{gen_server,call,3,[{file,"gen_server.erl"},{line,212}]},
         {rabbit_mgmt_util,'-amqp_request/5-fun-0-',4,
             [{file,"src/rabbit_mgmt_util.erl"},{line,579}]},
         {rabbit_mgmt_util,with_channel,5,
             [{file,"src/rabbit_mgmt_util.erl"},{line,598}]},
         {rabbit_mgmt_util,http_to_amqp,5,
             [{file,"src/rabbit_mgmt_util.erl"},{line,526}]},
         {webmachine_resource,resource_call,3,
             [{file,"src/webmachine_resource.erl"},{line,186}]},
         {webmachine_resource,do,3,
             [{file,"src/webmachine_resource.erl"},{line,142}]},
         {webmachine_decision_core,resource_call,1,
             [{file,"src/webmachine_decision_core.erl"},{line,48}]},
         {webmachine_decision_core,accept_helper,1,
             [{file,"src/webmachine_decision_core.erl"},{line,612}]}]}}

I have confirmed that I can connect to Postgres via psql -h db -U postgres

Can't see the messages

Hello I'm really interested in getting this working:

I've already installed rabbitMQ in Kubernetes with a Rabbitmq helm chart and modifying its values.yaml to include this plugin:

And it seems to up and running, but how can I see the Postgres messages that I'm sending with Notify?

I can catch the other messages but can't see (or don't know how) catch the PostgreSQL Notifications.

I'm using the prerelease version: v1.0.0a1

Compile for 3.6.x RabbitMQ

Please, update this plugin to support new version of RabbitMQ.