GithubHelp home page GithubHelp logo

Comments (6)

eulerto avatar eulerto commented on May 31, 2024

@dvarrazzo I agree that xids and lsn should have the same default. It is inconsistent. Do we favor replication system (those who will find this information useful) or system that expose data to some interface (those who doesn't care about transaction knowledge)?
There is a bug in the write-in-chunks code. It should emit valid JSON. It is useful for cases where you don't want to hold a huge transaction because it would consume a lot of server memory (see issue #4).

from wal2json.

dvarrazzo avatar dvarrazzo commented on May 31, 2024

@eulerto My personal view: I totally favour a system that doesn't care about the absolute identity of the transaction. I see this project as a nimble, clever plugin to get a stream of changes from a database for a receiver to do whatever manipulation it wants. People wanting a replication solution should rather use something more complete such as pglogical.

from wal2json.

eulerto avatar eulerto commented on May 31, 2024

@dvarrazzo I agree with your suggestion (include-xids = false) so I merged in #18. I'm not sure about write_in_chunks but decided to merge your suggestion because huge transactions are rare.

from wal2json.

rugging24 avatar rugging24 commented on May 31, 2024

@eulerto "... because huge transactions are rare" not exactly true. I come across those more frequently than you may imagine . Especially in cases where huge changes are made within a function. And these are essentially captured as a single transaction. Without the write-in-chunk feature of your plugin I wouldn't have been able to manage situations like that.
Just like @dvarrazzo mentioned "People wanting a replication solution should rather use something more complete such as pglogical.". In my case this is not used as a replication solution but as a change capture solution. So please reconsider different options before taking out that functionality .

from wal2json.

rugging24 avatar rugging24 commented on May 31, 2024

... plus when write-in-chunks is enabled , the format emitted is pretty consistent .
"{"change":[" at the beginning of the transaction and
"]}" at the end of it.
And everything in between are valid Json objects. One could easily find a work-around with this in ones client. Which is what I do in my case.

from wal2json.

eulerto avatar eulerto commented on May 31, 2024

@rugging24 Those features weren't removed. The default values were changed. From now on, if you want to include xids into 'changes' or that you want to write in chunks, you should explicitly inform that. For example:
pg_recvlogical -d postgres --slot test_slot --start -o include-xids=1 -o write-in-chunks=1 -f -

or via SQL:

SELECT data FROM pg_logical_slot_get_changes('test_slot', NULL, NULL, 'include-xids', '1', 'write-in-chunks', '1');

from wal2json.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.