PostgreSQL notification payload limit should not restrict job payload size about odd-jobs

[saurabhnanda]

Thank you for the PR @mrputty . It seems absolutely fine. In fact, in another place (unrelated to odd-jobs) where we are using LISTEN/NOTIFY for larger payloads, even we have faced this issue.

Just curious, what kind of large payloads are you putting in the job queue? Usually, we end-up putting the PK of another row in the DB as the job-payload (where the target row may be very large, not the job-payload itself).

Nevertheless, this optimization is good-to-have. I will merge it after compiling locally once.

[mrputty]

Great, glad to hear this will be merged.

My use case involves using odd-jobs to orchestrate the retrieval of individual items from a large data set. Within each item there are different subsets of data that may or may not need to be retrieved. The job-payload consists primarily of a list of item ids, tagged with the subsets that need to be retrieved. Some of these requests may include thousands of items, and we would quickly run into the 8000 byte limit, but we don't think they'll ever be terribly large.

Because the set of items to be retrieved does not have any meaning outside of the request, it seems to me that the overhead of maintaining them is in a secondary table is unnecessary, and also potentially difficult. For example, managing the lifetime of rows in the secondary table seems complicated. If one of these jobs fails, we would have to remember to delete the corresponding row (whose id would be embedded in the JSON job payload) when we were cleaning up the failed job records.

As I thought through managing this, I realized that there's also no way for me to enforce guarantees about record lifetimes even during normal processing. For example, if I were to write code to delete the secondary record at the end of job execution in my code, I have no way to guarantee that some other exception doesn't occur after the delete but before odd-jobs actually finalized the job. Since neither job runner nor any onJobStart callback have access to the underlying database connection, this cannot be done in a transaction.

Thinking about that was enough to encourage me to look for a simpler solution, which lead me to how pg_notify was being used.

[saurabhnanda]

@mrputty would you like to close and re-open a fresh MR, if you care about Hacktoberfest? Else I'll merge this one.