Enqueue problem (PostgreSQL 22P02) about river

[lexor]

I found the problem. I encounter this error when the default queue exec mode is selected as simple protocol or exec mode in the pgxpool config (config.ConnConfig.DefaultQueryExecMode = pgx.QueryExecModeSimpleProtocol). However pgx says DefaultQueryExecMode is incompatible with PGBouncer.

[brandur]

@lexor Nice find.

Did you see that PgBouncer now supports prepared statements if you're using transaction pooling as of a few months ago in version 1.21.0? There is a chance that pgx comment about PgBouncer support is outdated now ...

https://www.pgbouncer.org/faq.html#how-to-use-prepared-statements-with-transaction-pooling

[lexor]

@brandur Awesome. Thank you!

[brandur]

Odd, so the unique query isn't do anything too exotic. Basically just encoding JSON. Here it is:

			if params.UniqueByArgs {
				advisoryLockHash.Write([]byte("&args="))
				advisoryLockHash.Write(params.EncodedArgs)

				getParams.Args = params.EncodedArgs
				getParams.ByArgs = true
			}

Do you want to try doing a json.Marshal on your input ActivationEmailArgs, print in, and see if anything in there looks odd?

If not, maybe try removing each of the fields one at a time, and see if any one of them seems to be the offender? (It's probably email? But just a guess.)

[lexor]

@brandur I tried already removed all args with remove unique opts but facing same error (failed to insert job: ERROR: invalid input syntax for type json (SQLSTATE 22P02)). Also I tried with PostgreSQL 14.x and 16.x.

_, err := a.queue.Insert(ctx, worker.ExampleJobArgs{}, nil)
if err != nil {
   return fmt.Errorf("failed to insert job: %w", err)
}
...

type ExampleJobArgs struct {}

func (ExampleJobArgs) Kind() string {
	return "example"
}

func (ExampleJobArgs) InsertOpts() river.InsertOpts {
	return river.InsertOpts{}

type ExampleJobWorker struct {
	river.WorkerDefaults[ExampleJobArgs]
}

func (w *ExampleJobWorker) Work(
	ctx context.Context,
	job *river.Job[ExampleJobArgs],
) error {
	return nil
}

[bgentry]

Are you running this locally? I’m wondering if the Postgres server logs might reveal something about the value that’s causing the issue.

[lexor]

@bgentry Yes, running on docker. Here's my logs.

// "river.InsertOpts{UniqueOpts: river.UniqueOpts{ByArgs: true, ByPeriod: 4 * time.Hour}}"
2024-01-22 14:13:31.536 UTC [8745] ERROR:  invalid input syntax for type json at character 296
2024-01-22 14:13:31.536 UTC [8745] DETAIL:  Token "\" is invalid.
2024-01-22 14:13:31.536 UTC [8745] CONTEXT:  JSON data, line 1: \...
2024-01-22 14:13:31.536 UTC [8745] STATEMENT:  -- name: JobGetByKindAndUniqueProperties :one
	SELECT id, args, attempt, attempted_at, attempted_by, created_at, errors, finalized_at, kind, max_attempts, metadata, priority, queue, state, scheduled_at, tags
	FROM river_job
	WHERE kind = 'example'
	  AND CASE WHEN 't'::boolean THEN args = '\x7b7d' ELSE true END
	  AND CASE WHEN 't'::boolean THEN tstzrange('2024-01-22 12:00:00Z'::timestamptz, '2024-01-22 16:00:00Z'::timestamptz, '[)') @> created_at ELSE true END
	  AND CASE WHEN 'f'::boolean THEN queue = '' ELSE true END
	  AND CASE WHEN 't'::boolean THEN state::text = any('{available,completed,running,retryable,scheduled}'::text[]) ELSE true END

// "river.InsertOpts{}"
2024-01-22 14:14:25.253 UTC [8782] ERROR:  invalid input syntax for type json at character 225
2024-01-22 14:14:25.253 UTC [8782] DETAIL:  Token "\" is invalid.
2024-01-22 14:14:25.253 UTC [8782] CONTEXT:  JSON data, line 1: \...
2024-01-22 14:14:25.253 UTC [8782] STATEMENT:  -- name: JobInsert :one
	INSERT INTO river_job(
	  args,
	  attempt,
	  attempted_at,
	  created_at,
	  errors,
	  finalized_at,
	  kind,
	  max_attempts,
	  metadata,
	  priority,
	  queue,
	  scheduled_at,
	  state,
	  tags
	) VALUES (
	  '\x7b7d'::jsonb,
	  coalesce('0'::smallint, 0),
	  null,
	  coalesce('2024-01-22 14:14:25.252391Z'::timestamptz, now()),
	  null::jsonb[],
	  null,
	  'example'::text,
	  '3'::smallint,
	  coalesce('\x7b7d'::jsonb, '{}'),
	  '1'::smallint,
	  'default'::text,
	  coalesce(null::timestamptz, now()),
	  'available'::river_job_state,
	  coalesce(null::varchar(255)[], '{}')
	) RETURNING id, args, attempt, attempted_at, attempted_by, created_at, errors, finalized_at, kind, max_attempts, metadata, priority, queue, state, scheduled_at, tags
	

[brandur]

Odd, so the problem is that binary data being sent:

2024-01-22 14:14:25.253 UTC [8782] ERROR:  invalid input syntax for type json at character 225
2024-01-22 14:14:25.253 UTC [8782] DETAIL:  Token "\" is invalid.

Coming from '\x7b7d' which is {} in binary.

I can easily repro that:

platform_main_test=# select '\x7b7d'::jsonb;
ERROR:  invalid input syntax for type json
LINE 1: select '\x7b7d'::jsonb;
               ^
DETAIL:  Token "\" is invalid.
CONTEXT:  JSON data, line 1: \...

But I'd expect the encoded data to still be a non-binary {} at that point during insertion, and am not sure what would transform it that way. I'd guess something on the Go side of things?

[lexor]

@brandur I using Go 1.21.4. When I look at the river_job table for periodic jobs, I can see that their args column is added as "{}".

SELECT args, kind FROM river_job LIMIT 1;

| args | kind              |
|------|-------------------|
| {}   | example_periodic  |

[bgentry]

My hunch is that this is something that pgx is doing either in response to some config setting or environmental setting, or else a bug.

Can you confirm if you’re on the latest pgx v5 release?

If so I’m thinking we should be able to create a reproducible minimized example of this which only uses pgx with nothing layered on top.

[lexor]

@bgentry Yes latest pgx v5. Im using jackc/pgx/v5 v5.5.2.

[lexor]

I created a repository so that you can reproduce the error (https://github.com/lexor/river-args-reproduce).

[bgentry]

I’m wondering if there’s something we could/should be doing to check for invalid configurations like this. We could probably check this connection setting in the pgx pool upon driver init, for example. Thoughts @brandur ?

[brandur]

@bgentry Yes possibly.

I think before that though, we should verify that this isn't a pgx bug. A simple protocol seems to make sense to me, but why would that cause pgx to start sending jsonb as raw bytes when Postgres seems to clearly not support that?

[bgentry]

There’s a bit of a reference to something like this being impossible in this test comment https://github.com/jackc/pgx/blob/517c654e2cfa7f953bf3e7ac44fe2260cfde669a/stdlib/sql_test.go#L607

Want to ask Jack to confirm?