Real-time PostgreSQL: Blog entry, webinar, and documentation about citus

[sumedhpathak]

Below comment is from related Trello card.

We had a meeting today around the real-time dashboards use case (May 29, 2015).

One of the take aways from that meeting was providing a toolkit and/or instructions to handle table roll-ups in PostgreSQL. We discussed the following:

(1) For roll-up specifics, we discussed two alternatives:

(1a) Lazy materialized views with expiration times: http://hashrocket.com/blog/posts/materialized-view-strategies-using-postgresql

Andres noted that we can improve on this approach by blocking on the caller's side with a PG_TRY block.

(1b) Andres then noted that we could also create an extension, where we have a background worker process that does the "roll-up coordination".

We need to evaluate these two approaches, and pick one that we like better. Some points we chatted about the in the meeting were: Amazon RDS not being extension friendly, and if these approaches would be compatible with the approach we pick in CitusDB.

(2) Keep an invalidation queue or not

Kafka has become a standard for queueing. At the same time, it's an external dependency. How much effort is it to keep an invalidation queue table in Postgres?

For the motivation for this use-case, I'm copy+pasting from the email with the subject line "Real-time dashboards use case" dated May 21, 2015.

We had 5 customer interactions this week that targeted the same use-case: raw events data comes into the system, gets rolled up into cubes, and gets displayed through a NoSQL database.

As we chatted, if we can cover how users can address this use-case with CitusDB or PostgreSQL, that's a major win.

The current alternative pipeline we hear today are: (a) Redshift / Hadoop to store data, (b) Spark / MapReduce to transform the data, and (c) NoSQL database to serve the rolled up data.

For half of these conversations, the current INSERT rates were around 100/s. If we could document and provide a toolbox for this use case on PostgreSQL, their only objection will be that Postgres doesn't scale. We could then mitigate that by pointing them to the new CitusDB documents.

[ozgune]

We explored several methods to create roll-up tables and also presented a tutorial on it. We could evaluate our learnings from this tutorial and pick and implement an approach: https://www.youtube.com/watch?v=0ybz6zuXCPo

[marcocitus]

There are now many companies that successfully implemented a real-time analytics pipeline on Citus by creating rollup tables and using INSERT...SELECT.

Blog post: https://www.citusdata.com/blog/2018/06/14/scalable-incremental-data-aggregation/
Documentation: https://docs.citusdata.com/en/v7.4/use_cases/realtime_analytics.html