Can't connect to Greenplum server (psycopg2.ProgrammingError: column d.datcollate does not exist) about pgcli

[amjith]

This is now available in v1.0. Please upgrade.

pip install -U pgcli

[jackmaney]

Ah, I think I see the issue. This error occurs when trying to run the following query:

SELECT d.datname as "Name",
       pg_catalog.pg_get_userbyid(d.datdba) as "Owner",
       pg_catalog.pg_encoding_to_char(d.encoding) as "Encoding",
       d.datcollate as "Collate",
       d.datctype as "Ctype",
       pg_catalog.array_to_string(d.datacl, E'\n') AS "Access privileges"
    FROM pg_catalog.pg_database d
    ORDER BY 1;

(which grabs metadata about databases). The datcollate column exists in that table in PostgreSQL 9.4, but not in PostgreSQL 8.2.15 (which is the psql version upon which Greenplum is built).

[jackmaney]

The column seems to specify the LC_COLLATE configuration variable for a given database (which, along with the datctype column, specifies a locale for the database). Please let me know if I'm wrong, but I don't seem to see these columns used anywhere in the rest of the code. Are they necessary?

[amjith]

@jackmaney You're right about the dependency of Postgres 9.4. In order to mimic the special back-slash commands from psql, I had to translate a giant chunk of C-code from psql source to Python.

I chose to stick with the minimum viable implementation that supported the latest PG (it was 9.2 at the time). I'll work on adding graceful fallback when an older version is use.

[jackmaney]

Excellent. Thanks!

[gnmerritt]

Just wanted to chime in and mention that this unfortunately means pgcli won't work with Amazon's Redshift platform either. Would be happy to help try and add support if you'd like a hand and this issue isn't high priority

[amjith]

I would love to get some help on this one.

The code for dealing with the special command can be found here: https://github.com/amjith/pgcli/blob/master/pgcli/packages/pgspecial.py#L76

The code is a translation of C code from psql source code.

You can see the conditions that check for postgres versions in psql code here:
https://github.com/postgres/postgres/blob/master/src/bin/psql/describe.c#L1252

I don't know how to detect the version of Postgres using psycopg2, but it might not be necessary to detect the version if it can be wrapped in a retry loop if a command fails.

[jackmaney]

It's a fragile approach, but I know that select version(); produces a version string:

# select version();
                                                                              version
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 9.4.0 on x86_64-apple-darwin, compiled by i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00), 64-bit
(1 row)

[amjith]

Awesome! I forgot about select version().

I'll take fragile over non-existent. :)

[amjith]

Aha. There is a way to find the server version using psycopg2. http://initd.org/psycopg/docs/connection.html#connection.server_version

:)

[jackmaney]

Excellent. I'll try to scrape together a bit of time this weekend to give this a shot.

[hollobon]

Anyone working on this? Considering doing this myself but I don't want to duplicate effort.

[amjith]

@hollobon No one is working on it as far as I know. That's why I put a back_burner label on it. Because I couldn't find time to work on it.

If you need some help getting started please feel free to leave a comment.

[conorccamp]

I got this connected to Redshift (PostgreSQL 8.0.2) and am currently working on getting the meta commands working. I'll make a pull request this weekend but I'll need some help getting my code cleaned up since I'm still very much a beginner python developer.

[amelio-vazquez-reina]

Great news. 👍 Happy to help with beta testing or in any other way possible.

[amjith]

I'll be happy to help clean up the code. Feel free to open the PR.

The meta commands (or back-slash) command can be challenging to implement. So I'll be fine if you want to do that as a separate PR. :)

Thank you for taking a stab at it. :)

[hollobon]

I'm planning to translate the version-dependent bits from psql (mainly in src/tools/bin/psql/describe.c). Hoping to make a start tonight / tomorrow, but don't want to tread on toes so I'll hold off if others doing the same.

[jklukas]

I'd also like to use pgcli with Redshift, so +1 to this effort. I'm happy to participate in code review when a PR comes together.

[rakeshsingh]

+1
Using pgcli with Redshift, and getting the same error