ProgramLimitExceeded plugin wal2json about wal2json

[dmacthedestroyer]

I just ran into this issue as well. I was doing load testing and did a large (~800k rows) update.

I found this SO answer that implies we can't exceed 1GB for a single message (or something 🤷 I dunno what I'm doing).

Not sure what the solution is or should be (break the encoding into multiple parts?). I plan to monitor for this error and have some ugly manual intervention plan for when it occurs. I doubt I'll run into this scenario outside of my load testing exercises.

[eulerto]

@mohammadnaghibi @dmacthedestroyer If you are using format 1 (default), see #4 #46. You are probably have a big transaction. If you cannot avoid big transactions, use option --write-in-chunks. (Have in mind, that while the transaction is streamed the partial transaction is not a valid JSON. You should postpone the validation when it receives the whole transaction.)

However, I advise you to use format 2. It does not have the big transaction limitation.

In summary, format 1 writes the whole transaction to a single buffer (that is limited to 1 GB) and format 2 writes every change to a single buffer. Hence, format 2 might not fail for big transaction cases that used to fail in format 1.

There are still issues with a single change that needs more than 1 GB and there is no fix for it yet.