markdrago / pgsanity Goto Github PK

I have a file that contains mutiple statements and is partly commented out, like this:

select * from aap;

/*
select * from aap;
*/

on running:

from pgsanity.pgsanity import check_string
check_string(sql_command)

with a multi-line sql command I get:
*** Error in `ecpg': munmap_chunk(): invalid pointer: ***
I have found a workaround:

  sql_command = '\n'.join(
        filter(
            lambda line: line.strip() == "",
            sql_command.splitlines()
        )
    )

meaning I remove all empty lines.
I tried adding this snippet in: pgsanity.pgsanity.ecpg.check_syntax but it broke the tests:

  .............................F....F.FF..
======================================================================
FAIL: test_simple_failure (test.test_ecpg.TestEcpg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yonatan/github/pgsanity/test/test_ecpg.py", line 14, in test_simple_failure
    self.assertFalse(success)
AssertionError: True is not false

======================================================================
FAIL: test_check_invalid_string (test.test_pgsanity.TestPgSanity)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yonatan/github/pgsanity/test/test_pgsanity.py", line 34, in test_check_invalid_string
    self.assertFalse(success)
AssertionError: True is not false

======================================================================
FAIL: test_check_invalid_file (test.test_pgsanity.TestPgSanityFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yonatan/github/pgsanity/test/test_pgsanity.py", line 56, in test_check_invalid_file
    self.assertNotEqual(status_code, 0)
AssertionError: 0 == 0

======================================================================
FAIL: test_check_missing_semi (test.test_pgsanity.TestPgSanityFiles)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yonatan/github/pgsanity/test/test_pgsanity.py", line 65, in test_check_missing_semi
    self.assertNotEqual(status_code, 0)
AssertionError: 0 == 0

----------------------------------------------------------------------
Ran 40 tests in 0.020s

FAILED (failures=4)

how can I fix this?

Does not throw any errors for

echo "select from b;" | pgsanity

Can't get it running on centos 8, no ecpg in postgres-devel?

https://www.centos.org/forums/viewtopic.php?t=71898&p=302344

I'm going to look into how much of argparse I'd need to pull in to make this work on python 2.6.

Good morning and thank you for your amazin job.
I'm working with some sql files and scripts, full of variables and \command, and pgsanity flag every command as an error, is there any way to flag them not as an error?
I'm running pgsanity on a docker container, i'm using ur suggestions to replace every variable with a sed -e but i don't wanna use it with just a backslash because it could be a typo.
Thank you.

I'm currently syntax checking a 600mb SQL file and it'd be nice to know how far through pgsanity is.

Bonus: it's currently maxing out a single core of my VM and using 1.3GB of RAM :D

pgsanity 0.2.9
ecpg 10.1
Python 2.7.13
Windows 10

echo select column1 alias2 asdf from table3>example.sql

When I run pgsanity example.sql I get Try "ecpg --help" for more information..
(Running ecpg -o - - outputs the same message Try "ecpg --help" for more information..)

Maybe Windows is not supported?

ecpg, the program which pgsanity depends on, chokes on a unicode BOM (https://en.wikipedia.org/wiki/Byte_order_mark). Pgsanity should strip out unicode BOMs before sending the text off to ecpg.

Hello!

First of all, thank you for pgsanity -> it's a really nice piece of software 👍

I'm preparing a port of pgsanity for the FreeBSD ports at the moment. Would you find a minute please to create a 0.2.9 tag/release on GitHub so that it is easier to point to the latest version of pgsanity?

Thanks!

I just killed a long running pgsanity process after 3 days.
Would the following backtrace indicate that it was still in the python code?

Traceback (most recent call last):
File "/usr/local/bin/pgsanity", line 9, in
load_entry_point('pgsanity==0.2.4', 'console_scripts', 'pgsanity')()
File "/usr/local/lib/python2.7/dist-packages/pgsanity/pgsanity.py", line 68, in main
return check_files(config.files)
File "/usr/local/lib/python2.7/dist-packages/pgsanity/pgsanity.py", line 63, in check_files
accumulator |= check_file(filename, show_filenames)
File "/usr/local/lib/python2.7/dist-packages/pgsanity/pgsanity.py", line 33, in check_file
prepped_file = prep_file(filelike)
File "/usr/local/lib/python2.7/dist-packages/pgsanity/pgsanity.py", line 21, in prep_file
prepped_sql = sqlprep.prepare_sql(raw_sql)
File "/usr/local/lib/python2.7/dist-packages/pgsanity/sqlprep.py", line 9, in prepare_sql
for (start, end, contents) in split_sql(sql):
File "/usr/local/lib/python2.7/dist-packages/pgsanity/sqlprep.py", line 52, in split_sql
results = get_next_occurence(sql[start:], bookends)
File "/usr/local/lib/python2.7/dist-packages/pgsanity/sqlprep.py", line 70, in get_next_occurence
index = haystack.find(needle)
KeyboardInterrupt

The SQL I was using was from https://bitbucket.org/voltagex/acorndata - this is a very large repo, but have a look at the sql file in the master branch. This is completely invalid SQL as far as Postgres is concerned; I was hoping to use pgsanity to show me how much needed to be changed to migrate from MSSQL to PostgreSQL.

CREATE TABLE example_table (
  reference character varying
)

line 2: ERROR: syntax error at or near "reference"

In my big JavaScript projects, I like to run JSHint over all my .js code with:

$ jshint .

This automatically recurses over the current directory, or any other directory I pass in.

Could pgsanity please get this behavior as well?

As a workaround, I can do this:

find . -type f -name '*.psql' -exec pgsanity {} \;

But I'd prefer that pgsanity know to do this automatically.

First thanks for a great tool! Very useful ;)

The issue:
Valid postgres sql statements that contains "'#>" and "#>>" fails validation.

Postgres documentation for these operations: https://www.postgresql.org/docs/11/functions-json.html

Example:

bash-4.4# echo "select '{\"a\":[1,2,3],\"b\":[4,5,6]}'::json#>>'{a,2}'" | pgsanity
line 2: ERROR: syntax error at or near "#>>'{a,2}'
"

I often use Unix find to send many *.sql files individually to pgsanity:

$ find lib -name "*.sql" -exec pgsanity {} \;
line 7: ERROR: syntax error at or near ":db"
line 7: ERROR: syntax error at or near ":db"
line 7: ERROR: syntax error at or near ":db"

When I do this, pgsanity neglects to show the filename, making it hard to tell which files have the syntax errors. Please always show the filename, no matter how many files a pgsanity process is checking.

As a workaround, I'm using find lib -name "*.sql" \( -exec pgsanity {} \; -o -print \), but that prints the filenames after the warnings, rather than prefixing each line.

Valid sql function:

CREATE FUNCTION add(integer, integer) RETURNS integer
    AS 'select $1 + $2;'
    LANGUAGE SQL
    IMMUTABLE
    RETURNS NULL ON NULL INPUT;

Output from pgsanity:

$ pgsanity valid_function.sql
line 3: ERROR: syntax error at or near "SQL"

Link to relevant postgres documentation on functions

At least on my own system, the following passes with exit-code 0:

SELECT '
-- this is not really a comment' AS c;
SELECT '
-- neither is this' AS c spam;

Despite the fact the above statements should actually fail with something like:

psql:shouldfail3.sql:4: ERROR:  syntax error at or near "spam"
LINE 2: -- neither is this' AS c spam;

No error is emitted because the statement is incorrectly prepared as:

EXEC SQL SELECT '
-- this is not really a comment' AS c;
SELECT '
-- neither is this' AS c spam;

I presume that ECPG just ignores everything after the first semicolon above, and therefore the invalid SQL in this example is simply not encountered.

I discovered this when I was trying to understand how the chunking algorithm did its thing, having wanted to grok it before trying to implement the ability to step over psql-commands.

More realistic instances similar to the above scenario likely exist under use of $$.

Short of fully integrating a tokenizer to get around this, I could probably just up the rock-paper-scissors game that's already going on in prepare_sql() with the if not in_block_comment business. Doing so should allow psql-stuff to be handled more readily.

What's your take?

Any way I can contribute?

Hi! Noticed the following two issues when using pgsanity:

1. pgsanity seems to throw an error on the first line in this SQL code:

\set start 1234
delete from x where y >= :start;

2. Also doesn't seem to catch syntax errors in plpgsql code block:

DO $zzz$
dsafdsf
BEGIN
update test set id=4 where city='def';
END
$zzz$ LANGUAGE plpgsql;

When installing pgsanity either from github or using pip, I get the following error when running it:

Traceback (most recent call last):
File "/usr/bin/pgsanity", line 9, in
load_entry_point('pgsanity==0.2.5', 'console_scripts', 'pgsanity')()
File "/usr/lib/python3.4/site-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3.4/site-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python3.4/site-packages/pkg_resources.py", line 2088, in load
entry = import(self.module_name, globals(),globals(), ['name'])
File "/usr/lib/python3.4/site-packages/pgsanity/pgsanity.py", line 9, in
import sqlprep
ImportError: No module named 'sqlprep'

I'm running Arch Linux and I have found out that the problem disappears when I manually call the script with python2. However this isn't very usable, at least for someone new to python.

PostgreSQL uses a :colon syntax for parameterizing SQL commands with command line variables.

create-database.sql:

CREATE DATABASE :db;

Usage:

$ psql -f create-database.sql -v db=test

However, pgsanity/ecpg rejects these.

$ pgsanity create-database.sql
line 1: ERROR: syntax error at or near ":db"

Mark,

Since we now live in a world where Python 2 is unsupported, and your utility is the #1 stackoverflow result when googling for Postgres SQL validators... have you given any thought to making this work on Python 3, perhaps using the six library or something?

We'd probably deploy this as part of our test process if it were ported.

I am afraid that I do have a stupid elementary inquiry.
I am using Windows7 and do not know how to use it( like how to use linux sever ..etc).
I was wondering how ppl use pgsanity for Windows

OS:Windows7
Python:3.65

The documentation for the pre-commit hook (added in #35) currently states that rev: v0.2.9 should be used. However, the pre-commit hook was added after the v0.2.9 tag, which means that pre-commit fails to install the hook (since .pre-commit-hooks.yaml doesn't exist in the repo for tag v0.2.9).

Therefore, the documentation needs to be updated to point to either a new tag for the most recent version of the code (e.g. rev: v0.2.10) or the latest commit SHA (i.e. rev: 5fb3ce7).

Just tried to verify the sanity of below sql files

BEGIN

CREATE SCHEMA transactional;

CREATE TABLE transactional.passenger (
  --primary key
  passenger_type_id      SERIAL,

  --datas
  passenger_type_name    VARCHAR, 
  is_active              BOOLEAN,
  additional_data        JSONB,

  --metadata
  sys_update_description VARCHAR,
  sys_update_timestamp   TIMESTAMPTZ,
  sys_update_name        VARCHAR,
  sys_creation_name      VARCHAR DEFAULT 'CIMAHI',
  sys_creation_timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,

  PRIMARY KEY (passenger_type_id),
  UNIQUE (passenger_type_name)
);

-- other create table, omitted for succinctness

COMMIT;

And below is the result from pgsanity: line 3: ERROR: syntax error at or near "CREATE". After commenting line 3, the file passed the verification (even without commenting the last COMMIT; line).

Is there an issue? I am using Python 2.7.6.

Many thanks.