CSV filter not treating Buffer (bytea) correctly about pg-promise

[vitaly-t]

Sounds like perhaps you are not passing parameters correctly. Please include a reproducible code example here.

UPDATE

The issue you linked to is different, about Buffer escaping, while you are talking about CSV filter logic. That filter was never meant to be used for type Buffer, because that wouldn't make much sense. If your Buffer contains a password, just remove CSV filter, and the value will be escaped correctly.

[moltco]

You are partially correct, I was doing Buffer.from("abc").toString() with the $2:csv filter which has caused issues. If I do just Buffer.from("abc") this appears to be working. But the unicode null \u0000 definitely breaks the code and also I can't see any unicode string encapsulation when query is created (ie no E'abc').

Here is a code that will fail:

var obj = {
          id: crypto.randomUUID(), // ignore this - I need some fields for referential integrity/constratints
          org_id: id_org, // ignore 
          password: Buffer.from("test"), // ignore
          salt: Buffer.from("salt"), // ignore
          firstName: "Peter",  // ignore 
          lastName: '\uD83D\uDC04\u0000', // <- this fails
          email: '[email protected]' // ignore
        }

        const q = sql('generic/create.sql'); // INSERT INTO $1:alias($2:name) VALUES($2:csv) RETURNING *
        const insertResult = await db.one(q, ['users', obj]);
        console.log(JSON.stringify(insertResult));

With the above code I get error code is 08P01 on the lastName field.

If I change \u0000 to something else, e.g. \u0001 this works. However, when inspecting the text of the compiled query - there is no escaping for unicode characters or marking the string as unicode, e.g.E'\uD83D\uDC04\u0000' - I am not sure if this would improve things or not though but could be a prudent thing to do. I read on the interweb that for \u0000 to be passed to Postgress one needs more escaping so perhaps it's a special case that should be treated as such?

Here is the full error stack

invalid message format

INSERT INTO users("id","org_id","password","salt","firstName","lastName","email") VALUES('aa03c249-e82b-4e24-a759-29771d6e10d9','8e9ee4b1-65a1-464e-9d63-c67c29953fda','\x74657374','\x73616c74','Peter','🐄','[email protected]') RETURNING *

error: invalid message format
    at Parser.parseErrorMessage (C:\Users\m\source\repos\id\node_modules\pg-protocol\src\parser.ts:369:69)
    at Parser.handlePacket (C:\Users\m\source\repos\id\node_modules\pg-protocol\src\parser.ts:188:21)
    at Parser.parse (C:\Users\m\source\repos\id\node_modules\pg-protocol\src\parser.ts:103:30)
    at Socket.<anonymous> (C:\Users\m\source\repos\id\node_modules\pg-protocol\src\index.ts:7:48)
    at Socket.emit (C:\Users\m\source\repos\id\lib\events.js:517:28)
    at Socket.emit (C:\Users\m\source\repos\id\lib\domain.js:489:12)
    at addChunk (C:\Users\m\source\repos\id\lib\internal\streams\readable.js:335:12)
    at readableAddChunk (C:\Users\m\source\repos\id\lib\internal\streams\readable.js:308:9)
    at Socket.Readable.push (C:\Users\m\source\repos\id\lib\internal\streams\readable.js:245:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {length: 143, name: 'error', severity: 'ERROR', code: '08P01', detail: undefined, …}

[vitaly-t]

You can provide your own solution/work-around to the issue, via Custom Type Formatting, by escaping it as you want.

[moltco]

Thanks for the tip, I did consider CTFs (and use them for more complex types).

I was hoping that this issue would be recognized as a bug/issue and a patch would go into the pg-promise. The \u0000 in a string is a valid string (albeit an edge case) for javascript so, ideally, I shouldn't have to create a workaround for the pg-promise, it should 'just work'.

[vitaly-t]

The original issue was never resolved, and I do not see what solution, if any, is possible inside this library. Most likely, this needs to be addressed inside the base driver, and not here.

[vitaly-t]

I am closing it, because I'm not planning to do any fixes on this end, it is down to the base driver, as per the old linked issue.