quix-labs / pg-el-sync

Synchronize yout PostgreSql Data into Elasticsearch in realtime

Introduction

This tool is designed to synchronize data between a Postgresql database and Elasticsearch. It uses a trigger system with a pub/sub mechanism to monitor real-time changes in the database and index them in Elasticsearch.

Installation and Configuration

Using Docker

• Pull the Docker image from the GitHub Container Registry:

  docker pull ghcr.io/quix-labs/pg-el-sync:latest

Using Go

• Install the tool using Go:

  go install github.com/quix-labs/pg-el-sync@latest

Using prebuilt assets

You can also install the tool using release assets such as .deb, .apk, or others.

Download the appropriate package from the Releases page, and then follow the instructions provided for your specific platform.

Usage

The tool provides two main commands for usage:

• pg-el-sync listen: Start listening to the PostgreSQL database for real-time changes and sync them with Elasticsearch.
• pg-el-sync index: Index all tables from the PostgreSQL database into Elasticsearch.

Using Docker

Listen Command

docker run -v /path/to/config.yaml:/app/config.yaml ghcr.io/quix-labs/pg-el-sync:latest pg-el-sync listen

Index Command

docker run --rm -v /path/to/config.yaml:/app/config.yaml ghcr.io/quix-labs/pg-el-sync:latest pg-el-sync index

Configuration

Configuration file

By default pg-el-sync load configuration file from /app/config.yaml, you can override using environment variable:

  export CONFIG_FILE=/path/to/config.yaml
  # Or with docker: docker run -e CONFIG_FILE=/path/in/docker/config.yaml ...

Supervisord Configuration

To manage the pg-el-sync listen command with Supervisord, you can use the following configuration:

[program:pg-el-sync]
command=/path/to/pg-el-sync listen 
autostart=true
autorestart=true
startsecs=10
startretries=3
stdout_logfile=/var/log/pg-el-sync-listen.log
stderr_logfile=/var/log/pg-el-sync-listen.err.log

Make sure to replace /path/to/pg-el-sync with the absolute path to your pg-el-sync installation.

You can also adjust other settings according to your needs, such as log file paths and startup parameters.

Local Development

To set up the environment for local development, follow these steps:

1. Clone this repository:

   git clone https://github.com/quix-labs/pg-el-sync

2. Navigate to the pg-el-sync directory:

   cd pg-el-sync

3. Copy the config.example.yaml file to config.yaml and configure it according to your environment:

   cp config.example.yaml config.yaml

4. Run the tool in development mode:

   go run .

Credits

• COLANT Alan
• All Contributors

License

This project is licensed under the MIT License.

pg-el-sync's Issues

Ajout parametre lazy sur relation et split dans une nouvelle materialized view

DROP MATERIALIZED VIEW documentsview;
CREATE MATERIALIZED VIEW documentsview AS 
SELECT id as reference,annonce_id as annonce_id, JSON_BUILD_OBJECT('order',"documents"."order",'url',CASE WHEN documents.disk='media_images' THEN CONCAT('https://my-cdn.com/',documents.name) WHEN documents.disk='manual' THEN CONCAT('https://mymanual-cdn.com',documents.name) END)
FROM documents;
---Annonces---
CREATE MATERIALIZED VIEW annoncesview AS
SELECT annonces.id as reference, JSON_BUILD_OBJECT('documents',JSON_AGG(documentsview.*))
FROM annonces
LEFT OUTER JOIN documentsview ON documentsview.annonce_id = annonces.id
GROUP BY annonces.id;

Plugin system

Add ability to modify documents between PostgreSql reponse and indexing

• Create new config column for indices:

plugins:
  - name: myplugin
    language: golang # OR bash

• Allow multiple language golang/bash
• Allow redirecting document to delete/update/create channel manually

Fix pivot trigger when reference is string