hasura / skor

Now part of Hasura GraphQL Engine. Listen to postgres events and forward them as JSON payloads to a webhook

Skor

New and improved version of Skor is now part of Hasura GraphQL Engine

A few months ago, we built the open source GraphQL Engine that gives you instant GraphQL APIs over any Postgres database. We have added all of Skor's existing features and even more to make it production ready:

1. Reliable: We capture every relevant action on the database as an event, even when Hasura is down! The events are delivered to your webhook as soon as possible with an atleast-once guarantee.

2. Scalable: What more, it even scales horizontally. If you are processing millions of events, just add more instances of GraphQL engine.

3. Use with Serverless: If you are using Skor, then avoid the pain of managing your webhook by moving to Serverless infrastructure. Check out these blog posts to get started

Use Hasura GraphQL Engine for production use cases

---

skor is a utility for Postgres which calls a webhook with row changes as JSON whenever an INSERT, UPDATE or DELETE event occurs on a particular table. You can drop the docker image next to your Postgres database instance and configure a webhook that will be called.

It works using a pg_notify trigger function and a tiny C program skor that listens to the notifications and calls the configured webhook with a JSON payload.

When to use

• When you want to trigger an action in an external application when a table row is modified.
• When you want a lightweight notification system for changes in the database.
• When you want to send the changes to a message queue such as AMQP, Kafka etc.

How it works

A PostgreSQL stored procedure is set up as a trigger on the required table(s). This trigger uses PostgreSQL's LISTEN and NOTIFY to publish change events as JSON to a notification channel. Skor watches this channel for messages and when a message is received, it makes an HTTP POST call to the webhook with the JSON payload. The webhook can then decide to take an action on this.

Caveats

• Events are only captured when skor is running.
• If a call to the webhook fails, it is not retried.

Getting started

1) Set up the triggers:

We need to setup triggers on the tables that we we are interested in. Create a triggers.json file (see sample.triggers.json) with the required tables and events.

Note: This command requires python3.

$ ./gen-triggers.py triggers.json | psql -h localhost -p 5432 -U postgres -d postgres --single-transaction --

2) Run Skor:

Run the skor Docker image (that has the skor binary baked in):

$ docker run \
    -e DBNAME="postgres" \
    -e PGUSER="postgres" \
    -e PGPASS="''" \
    -e PGHOST="localhost" \
    -e PGPORT=5432 \
    -e WEBHOOKURL="http://localhost:5000/" \
    --net host \
    -it hasura/skor:v0.1.1

Make sure you use the appropriate database parameters and webhook URL above.

Examples

INSERT

Query:

INSERT INTO test_table(name) VALUES ('abc1');

JSON webhook payload:

{"data": {"id": 1, "name": "abc1"}, "table": "test_table", "op": "INSERT"}

UPDATE

Query:

UPDATE test_table SET name = 'pqr1' WHERE id = 1;

JSON webhook payload:

{"data": {"id": 1, "name": "pqr1"}, "table": "test_table", "op": "UPDATE"}

DELETE

Query:

DELETE FROM test_table WHERE id = 1;

JSON webhook payload:

{"data": {"id": 1, "name": "pqr1"}, "table": "test_table", "op": "DELETE"}

Uninstalling

To remove the skor related functions and triggers that were added to Postgres, run this in psql:

DO $$DECLARE r record;
BEGIN
    FOR r IN SELECT routine_schema, routine_name FROM information_schema.routines
             WHERE routine_name LIKE 'notify_skor%'
    LOOP
        EXECUTE 'DROP FUNCTION ' || quote_ident(r.routine_schema) || '.' || quote_ident(r.routine_name) || ' CASCADE';
    END LOOP;
END$$;

Deploying Skor on Hasura

The pre-built Docker image with the skor binary is available at hasura/skor and can be deployed as a microservice with the sample k8s.yaml in this repo. The webhook can be another microservice that exposes an endpoint.

To learn more on deploying microservices on Hasura you may check out the documentation.

Build Skor:

Requirements:

• PostgreSQL 9+
• gcc
• libcurl (libcurl4-openssl-dev)
• libppq (libpq-dev)

Build:

$ make

Run:

$ ./build/skor 'host=localhost port=5432 dbname=postgres user=postgres password=' http://localhost:5000

Test

1. Install the requirements specified in tests/requirements.txt
2. The tests assume that you have a local postgres instance at localhost:5432 and a database called skor_test which can be accessed by an admin user.
3. Run skor on this database with the webhook url set to http://localhost:5000
4. run run_tests.sh script in the tests directory.

Contributing

Contributions are welcome!

Please check out the contributing guide to learn about setting up the development environment and building the project. Also look at the issues page and help us in improving Skor!

skor's Issues

Query generated by gen-triggers.py failed on Postgres 9.6

When attempting to execute the query I am receiving the following error on Postgres 9.6

ERROR:  syntax error at or near "CASCADE"
LINE 1: ...NCTION public.notify_skor_github_repositories_insert CASCADE
                                                                ^
QUERY:  DROP FUNCTION public.notify_skor_github_repositories_insert CASCADE
CONTEXT:  PL/pgSQL function inline_code_block line 6 at EXECUTE

It executes correctly on Postgres 10 but unfortunately am stuck on 9.6 in production so cannot install the triggers correctly.

Is it just not supported or can the query be modified to be compatible?

Deployment instructions

General deployment instructions with Docker
Deployment instructions with Hasura

Release checklist

• change binary name
• write test:
• set up env
• run tests
• teardown
• When to use and sample use-cases
• Webhook and payload documentation for each type of webhook
• How to use on Hasura
• Diagram of how it works
• Contributors/maintainers section

document the customisation of the trigger

For example it is necessary with tables with large rows.

create a per table and specific columns webhook

We maybe interested in changes to a specific column or columns in a specific table and not all db events

Skor Questions

👋🏽 Hello!

First, what a great project! I love Postgres' LISTEN/NOTIFY feature and am using it as a key part of a system I'm currently building.

I have two questions:

1. Postgres has an 8000 byte limit on message payload size, and an 8 GB queue size limit for all notifies (ref: https://www.postgresql.org/docs/9.6/static/sql-notify.html). It's possible to hit the first limit very easily with a normal-sized jsonb column, and I'm not sure how many instances out there run with less than 8 GB of memory, but in the case that either limit is reached, how does Skor deal with it? Does it just pass the error from Postgres along to the caller or crash?

2. Since notifications have no access control, meaning any code that can connect to your database can send notification messages, message payloads can't really be trusted. In my own system, I only use them to notify the caller that something has changed. I looked briefly at https://pgxn.org/dist/pg_message_queue/, but ultimately decided to just grab the changed row when notified, instead. How does Skor deal with this issue?

Thanks!

document 'per statement' triggers

See the discussion at https://news.ycombinator.com/item?id=16976459

Accept database connection string

@0x777

Add a init script that drops and adds triggers given table names

When setting up triggers for my tables I want to do something like this:

psql -U admin -d hasuradb < $(init.sh table1,table2,table3)

init.sh is a bash script that takes multiple arguments or a comma separated list of tables names and generates the DROP IF EXISTS CREATE TRIGGER kind of sql that I can pipe into psql.

This is very important as the schema evolves.

service of skor can connect with database but don't response when insert data to postgres

Hi i have problem about response web hook

i can build skor.c and run with command "./build/skor 'host=localhost port=5432 dbname=postgres user=postgres password=' http://localhost:5000 " and then i try insert data to postgres but
don't response when insert data to postgres

i don't know that what i did wrong

rate limiting/retries of webhook calls

Queries made using psycopg2 don't fire triggers

Reason unknown. Hence used a subprocess.Popen psql hack to run tests.

Deploy on heroku button, and git push to deploy on heroku, use with graphql-engine

New and improved Skor is now part of Hasura GraphQL Engine

New and improved version of Skor is now part of Hasura GraphQL Engine

A few months ago, we built the open source GraphQL Engine that gives you instant GraphQL APIs over any Postgres database. We have added all of Skor's existing features and even more to make it production ready:

1. Reliable: We capture every relevant action on the database as an event, even when Hasura is down! The events are delivered to your webhook as soon as possible with an atleast-once guarantee.

2. Scalable: What more, it even scales horizontally. If you are processing millions of events, just add more instances of GraphQL engine.

3. Use with Serverless: If you are using Skor, then avoid the pain of managing your webhook by moving to Serverless infrastructure. Check out these blog posts to get started

Use Hasura GraphQL Engine for production use cases