Comments (6)
See commit dc6e390.
wal2json nextlsn is different from pg_logical_slot_*_changes lsn. lsn assumes different positions depending on the WAL record type (first_lsn for begin, end_lsn for commit, and lsn for changes). See details in backend/replication/logical/logical.c (functions begin_cb_wrapper, commit_cb_wrapper, and change_cb_wrapper).
from wal2json.
Hi
Sorry for the late reply.
Does it mean that each WAL record doesn't have a single LSN but multiple? (first_lsn, lsn, end_lsn) i.e. a single WAL record spans multiple LSNs?
I'm not clear how the LSN numbering works. Can you explain a bit more about it?
Also wal2json gives the nextlsn which is apparently the lsn of the next record (which is the same as end_lsn of the current record?). I can use this nextlsn value as a starting point for resuming consumption.
But what value do I use to mark as the flushed lsn value to let postgres know that the client has read upto the current record?
If I use the nextlsn value, won't it mean that the client has read upto next record whereas it actually has read only upto the current record (something like current_lsn)?
from wal2json.
Does it mean that each WAL record doesn't have a single LSN but multiple? (first_lsn, lsn, end_lsn) i.e. a single WAL record spans multiple LSNs?
WAL is a stream of changes. LSN stands for Log Sequence Numbers i.e. a position in the WAL stream whose unit is bytes. When I mentioned first_lsn, lsn, and end_lsn I was refering to the logical replication context which use those "marks" to point to start of the transaction, a command in the transaction, and end of the transaction, respectively.
I'm not clear how the LSN numbering works. Can you explain a bit more about it?
Don't get me wrong, read src/backend/access/transam/README.
Also wal2json gives the nextlsn which is apparently the lsn of the next record (which is the same as end_lsn of the current record?). I can use this nextlsn value as a starting point for resuming consumption.
But what value do I use to mark as the flushed lsn value to let postgres know that the client has read upto the current record?
I suggest that you use replication origin instead of inventing your own replication progress tracking solution.
If I use the nextlsn value, won't it mean that the client has read upto next record whereas it actually has read only upto the current record (something like current_lsn)?
nextlsn points to the end of the commit record. That is what a replication progress tracking solution should store to restart stream in case of crash (if you search for end_lsn in src/backend/replication/logical/*.c you'll understand what I'm trying to explain).
from wal2json.
Commenting to say that I utilized this in one of my projects, PGDeltaStream š
Thanks!
from wal2json.
Could I pick this discussion back up for a related question? I notice that sometimes the first record I receive after attaching to an existing logical replication slot carries nextlsn: 0/0
$ pg_recvlogical -p 33107 -h localhost -U postgres -S slot -d postgres --start -f - -o include-lsn=1 -v
pg_recvlogical: starting log streaming at 0/0 (slot slot)
pg_recvlogical: streaming initiated
pg_recvlogical: confirming write up to 0/0, flush to 0/0 (slot slot)
{"nextlsn":"0/0","change":[{"kind":"insert","schema":"public","table":"test","columnnames":["col1","col2"],"columntypes":["character varying","integer"],"columnvalues":[null,104]}]}
pg_recvlogical: confirming write up to 0/173A488, flush to 0/173A488 (slot slot)
This is a slot that has received and acknowledged records through the same pg_recvlogical command before and the database was not at 0/0.
This is not a bug report btw and I understand wal2json just echoes what's in the ReorderBufferTXN, but I was wondering if either of you two could shed some light on why I'm seeing 0/0?
from wal2json.
Regarding replication origins: people say that to use them, one needs some superuser permissions which is not compatible with e.g. AWS RDS. So sometimes manual tracking is still beneficial.
Also, would be cool to have some āvanilla pseudocodeā example in wal2jsonās readme which shows, how exactly should the messages be processed/consumed and when should they be acked (especially in case of write-in-chunks mode). Itās non trivial, people make mistakes here and then even live with them for days/months until it appears that their solutions donāt fully work. I see many questions here and on StackOverflow which couldāve just been answered all at once with such a pseudocode.
from wal2json.
Related Issues (20)
- PostgreSQL 15 support HOT 1
- wal2json limitation HOT 2
- Can't enable truncation for format-version 1 HOT 1
- WAL2JSON Format-1 not including Current lsn HOT 3
- When attempting logical decoding : /usr/lib64/pgsql/wal2json.so: undefined symbol: elog_finish HOT 2
- Change data not captured properly while operating with partition tables HOT 1
- Getting table name in double Quotes HOT 2
- Segmentation fault HOT 1
- ERROR: could not load library "C:/Program Files/PostgreSQL/9.5/lib/wal2json.dll": The specified module could not be found. HOT 1
- START_REPLICATION command does not work with wal2json options HOT 1
- NaN values are received as "null" for NUMERIC type while capturing change data HOT 4
- Invalid JSON with non-transactional message HOT 5
- WAL record received in different formats when the table name contains the single quote HOT 1
- Add support for update_replication_progress introduced in pg15
- Install failed on Alpine 15 HOT 2
- ProgramLimitExceeded plugin wal2json HOT 2
- include-pk can't work with identity full? HOT 2
- Official instructions to build from source for production use HOT 1
- Change data not captured HOT 3
- LSN not picking from "withStartPosition" in format version-2 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
š Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ššš
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ā¤ļø Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wal2json.