Scripts using postgresql dollar-quoted strings fail in version 1.5.1 about grate

[erikbra]

Fixed in Release 1.5.2

[Westat-Transportation]

@koenekelschot sent repro file via email.

[Westat-Transportation]

I am seeing the same error while trying to run an Up script created using pg_dump that includes dollar-quoted strings.

Interestingly enough I get another error when trying to use a script created by pg_dump with the --disable-dollar-quoting option.

    Severity: ERROR
    SqlState: 42601
    MessageText: unterminated quoted string at or near "'"
    Position: 1342
    File: scan.l
    Line: 1145

[Westat-Transportation]

Reverting to grate 1.5.0 makes it so the dollar-quoted dump restores successfully.

[koenekelschot]

@Westat-Transportation is there any chance you can share the script with dollar quoting disabled? I could use that as a test case for my proposed fix.

[Westat-Transportation]

Sure, can I send it privately? It's for a LOB system db and its schema is proprietary.

[koenekelschot]

I'm not sure if you can see the email address in my profile. But I don't need the whole script, I suspect there is a query using multiple single quotes (as an escape). As in this contrived test case: https://github.com/erikbra/grate/pull/358/files#diff-1527405d352f46baa59fd9735ebf1a6996447c0b3ccf3badad7bdb5d5c49615a (line 41).
That's the part I hope to verify.

[Westat-Transportation]

Folks, gave grate 1.5.2 a go today and hit similar crashes on $$ terminated functions. The error message mentioned a line where $$ was declared and immediately followed by another keyword:

Error running script "001-db_schema.sql": 42601: unterminated dollar-quoted string at or near "$$TRUNCATE TABLE gis.tps_tbw_gps_places"

So I edited the .sql file and made sure all of the AS $$ declarations were followed by a line break with the next statement on the next line. When I did that I got another error:

Error running script "001-db_schema.sql": 42601: unterminated quoted string at or near "'"

Went back to 1.5.0 and it worked with both versions of the file 001-db_schema.sql. So I think there may still be room for improvement in parsing files using $$ termination. I will email the git repo which is running these separately -- that will allow you to see my edits from this morning.

[Westat-Transportation]

I am doing some local debugging against my files here. Will update this later if I can make it work.

[Westat-Transportation]

No luck, my attempts at tweaking the RegEx expressions did not work. I modified the code to write out the parsed statements so I can see where it was on the file when it bombed and found out that the following conditions were problematic for it:

• semicolons inside single quotes as part of text concatenation expressions (col_name || '; '::text).
• DO code blocks delimited by $do$

[Westat-Transportation]

@erikbra and @koenekelschot, I decided to take my modified code which records to a separate file the actual split statements that were sent to the db (just now realized I could have change verbosity, oh well) and bring over what the RegExes were in 1.5.0 and realized that grate was not really splitting PostgreSQL statements back in 1.5.0. The whole contents of a database dump were being sent to the server via npgsql.

Given the difficulties we are having to get this splitting to work in 1.5.1 and 1.5.2 what about introducing a CLI switch which basically turns that off on grate? I realize this mode would not work for cases like bugs/334-pgsql-concurrent. But it would allow folks like me to keep using the latest until a better approach for statement splitting can be developed.

Just a thought.