Comments (6)
@dvarrazzo I agree that xids and lsn should have the same default. It is inconsistent. Do we favor replication system (those who will find this information useful) or system that expose data to some interface (those who doesn't care about transaction knowledge)?
There is a bug in the write-in-chunks code. It should emit valid JSON. It is useful for cases where you don't want to hold a huge transaction because it would consume a lot of server memory (see issue #4).
from wal2json.
@eulerto My personal view: I totally favour a system that doesn't care about the absolute identity of the transaction. I see this project as a nimble, clever plugin to get a stream of changes from a database for a receiver to do whatever manipulation it wants. People wanting a replication solution should rather use something more complete such as pglogical.
from wal2json.
@dvarrazzo I agree with your suggestion (include-xids = false) so I merged in #18. I'm not sure about write_in_chunks but decided to merge your suggestion because huge transactions are rare.
from wal2json.
@eulerto "... because huge transactions are rare" not exactly true. I come across those more frequently than you may imagine . Especially in cases where huge changes are made within a function. And these are essentially captured as a single transaction. Without the write-in-chunk feature of your plugin I wouldn't have been able to manage situations like that.
Just like @dvarrazzo mentioned "People wanting a replication solution should rather use something more complete such as pglogical.". In my case this is not used as a replication solution but as a change capture solution. So please reconsider different options before taking out that functionality .
from wal2json.
... plus when write-in-chunks is enabled , the format emitted is pretty consistent .
"{"change":[" at the beginning of the transaction and
"]}" at the end of it.
And everything in between are valid Json objects. One could easily find a work-around with this in ones client. Which is what I do in my case.
from wal2json.
@rugging24 Those features weren't removed. The default values were changed. From now on, if you want to include xids into 'changes' or that you want to write in chunks, you should explicitly inform that. For example:
pg_recvlogical -d postgres --slot test_slot --start -o include-xids=1 -o write-in-chunks=1 -f -
or via SQL:
SELECT data FROM pg_logical_slot_get_changes('test_slot', NULL, NULL, 'include-xids', '1', 'write-in-chunks', '1');
from wal2json.
Related Issues (20)
- Getting table name in double Quotes HOT 2
- Segmentation fault HOT 1
- ERROR: could not load library "C:/Program Files/PostgreSQL/9.5/lib/wal2json.dll": The specified module could not be found. HOT 1
- START_REPLICATION command does not work with wal2json options HOT 1
- NaN values are received as "null" for NUMERIC type while capturing change data HOT 4
- Invalid JSON with non-transactional message HOT 5
- WAL record received in different formats when the table name contains the single quote HOT 1
- Add support for update_replication_progress introduced in pg15
- Install failed on Alpine 15 HOT 2
- ProgramLimitExceeded plugin wal2json HOT 2
- include-pk can't work with identity full? HOT 2
- Official instructions to build from source for production use HOT 1
- Change data not captured HOT 3
- LSN not picking from "withStartPosition" in format version-2 HOT 2
- Building dll for windows HOT 4
- Build and Install In RDS
- wal2json_16 is not available in CentOS7 HOT 1
- Unable to compile wal2json on Mac M1 HOT 1
- Installcheck in a loop eventually fails
- Best practice to handle "no old tuple data for UPDATE in table"? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wal2json.