semiocast / pgsql Goto Github PK

Hi,

in my code I do a copy like this (Connection is the PID of a pgsql_connection process):

copy_to(TableName, Data, Connection) ->
    {copy_in, [text]} = pgsql_connection:simple_query(["COPY \"", TableName, "\"(time, key, value) FROM STDIN"], Connection),
    ok                = pgsql_connection:send_copy_data(Data, Connection),
    {copy, Count}     = pgsql_connection:send_copy_end(Connection),
    Count.

Since the Connection is shared by several client processes, there can be some interference here, resulting in errors like "unexpected message type 0x51".

Would it make sense to wrap this three calls into one so they are always executed in sequence?
I can implement this, just wanted to check if this makes sense or if I am using it incorrectly.

Hi,

A project I'm helping (sumo_db) is considering the use of pgsql. More here

However, there is no license file. And that makes our position a bit difficult (yes I know the README has it). And it will be even better if you made one.

Thanks.

I'm having trouble inserting data into a bytea[] column (ARRAY::bytea), e.g for a table defined as:

CREATE TABLE example (
   id                bytea,
   tpe               text,
   lst               bytea[]
);

The insert Query is

INSERT INTO example (id, tpe, lst) VALUES($1::bytea, $2, $3::bytea[]);

Using pgsql_connection:extended_query(Query, Args, Conn) with

Args=[<<6,33,214,129,16,2,118,52,20,216,86,186,79,231,189,86>>,<<"b2b">>,[<<28,13,60,119,163,84,92,94,204,149,97,5,132,202,17,213>>]]

Gives the error

{error,{pgsql_error,[{severity,<<"ERROR">>},
                     {{unknown,86},<<"ERROR">>},
                     {code,<<"22021">>},
                     {message,<<"invalid byte sequence for encoding \"UTF8\": 0xa3">>},
                     {file,<<"wchar.c">>},
                     {line,<<"2017">>},
                     {routine,<<"report_invalid_encoding">>}]}}

Not sure if the query is not correct or this is a driver encoding issue. Any ideas?
Thanks!

Any interest in publishing pgsql to Hex? It's a little more heavyweight than just tagging the Github releases, but I've been enjoying using Hex dependencies with Rebar3.

The Rebar tutorial on publishing to Hex: http://www.rebar3.org/v3.0/docs/publishing-packages

The current decode_value_bin converts UUIDs to their hex representation: in addition to not being a very efficient implementation of the conversion process (io:format and then list_to_binary), both of the Erlang UUID libraries I know of use 16-byte binaries as their native storage format; so, if you have code that internally uses UUIDs for anything, you likely end up finding yourself taking the hex representation being returned by pgsql and running it back through a UUID parser to get the actual/normal binary representation. If it isn't considered possible to just change this (as it would, of course, break existing users who are currently re-parsing the hex UUIDs), maybe this could be made an option? ;P (Does the library maybe do this because the text representation of a UUID happens to be the hex representation? I'd personally rather see those get parsed back to the binary format--again, for better compatibility with Erlang UUID libraries--than the other way around.)

As a basic example,

SELECT NULL returns {{select,1},[{null}]}
but SELECT array[NULL, NULL]::varchar[] returns {{select,1},[{{array,[<<"NULL">>]}}]}

Another one:
select ARRAY['2014-01-01T12:12:12Z', NULL]::timestamp[]; will crash the process with

{{badmatch,{error,{fread,unsigned}}},[{pgsql_protocol,decode_value_text,3,[{file,"src/pgsql_protocol.erl"},{line,813}]},{pgsql_protocol,decode_array_text,4,[{file,"src/pgsql_protocol.erl"},{line,940}]},{pgsql_protocol,decode_array_text

The actual use case where I came across this was with two similar queries (the only difference is the where clause):

SELECT
            uuid,
            start_time,
            end_time,
            state,
            phone_number,
            direction,
            agent,
            array_agg(code) as wrap_up_codes,
            queue,
            array_agg(skill) as skills
        FROM phone_call
        LEFT JOIN phone_call_skill on phone_call.id=phone_call_skill.phone_call_id
        LEFT JOIN phone_call_wrap_up_code on phone_call.id=phone_call_wrap_up_code.phone_call_id GROUP BY id ORDER BY start_time DESC LIMIT 1

Returns
[{<<"73a13a1f5d78697568a4fd58031aa6ea3b30ce43">>,{{2015,5,13},{19,24,55}},null,<<"waiting">>,<<"5552221111">>,{call_direction,<<"inbound">>},null,{array,[<<"NULL">>]},<<"appointment">>,{array,[<<"NULL">>]}}]

whereas

SELECT
            uuid,
            start_time,
            end_time,
            state,
            phone_number,
            direction,
            agent,
            array_agg(code) as wrap_up_codes,
            queue,
            array_agg(skill) as skills
        FROM phone_call
        LEFT JOIN phone_call_skill on phone_call.id=phone_call_skill.phone_call_id
        LEFT JOIN phone_call_wrap_up_code on phone_call.id=phone_call_wrap_up_code.phone_call_id WHERE uuid = $1 GROUP BY id ORDER BY start_time DESC LIMIT 1

Returns [{<<"73a13a1f5d78697568a4fd58031aa6ea3b30ce43">>,{{2015,5,13},{19,24,55}},null,<<"waiting">>,<<"5552221111">>,{call_direction,<<"inbound">>},null,{array,[null]},<<"appointment">>,{array,[null]}}]

Thanks!

Is there a way to run the application using Makefiles instead of rebar?

I'm doing a simple_query and the result has a postgres timestampz type that include the timezone +/- at the end. This - is messing with the type conversion of the result.

Result it is choking on a row like this:

(1,XXXXXXXXX,1,ZZZZ,,"2014-08-29 10:34:25-06","2014-08-29 10:34:25-06")

Example stacktrace

      [{erlang,list_to_integer,["25-06"],[]},
       {pgsql_protocol,decode_value_text,3,
           [{file,
                "...pgsql/src/pgsql_protocol.erl"},
            {line,817}]},
       {pgsql_protocol,decode_row0,5,
           [{file,
                "...pgsql/src/pgsql_protocol.erl"},
            {line,739}]},
       {pgsql_connection,pgsql_simple_query_loop,4,
           [{file,
                "...pgsql/src/pgsql_connection.erl"},
            {line,769}]}]},

I forked and was about to switch src/pgsql.app to src/pgsql.app.src and then noticed your rebar.config.script.

I'm now curious why you did this? And if you want to just have pgsql.app why not simply put it in ebin/ then rebar will work find with it without the script file.

I'm pretty new to erlang so this may just be me, but I find the following rather confusingly worded:

For convenience with other database APIs, connection objects can be used as parametrized modules as connection objects are {pgsql_connection, pid()} tuples.

Does it mean this?

For convenience with other database APIs, connection objects can be used as parametrized modules. Connection objects are {pgsql_connection, pid()} tuples.

I'd be happy to help if I knew it'd get merged. At mo, number of unmerged PRs and issues makes it looks like the project is abandoned.

Please publish a new release to hex, there have been a number of useful features and fixes since the last publish.

Might sound a little bit stupid, but is there any reliable way to be notified when a pgsql_connection get disconnected. I have tried erlang:monitor/2 in combination with {reconnect, false}. But in cases where postgres terminate abruptly, the pgsql_connection process doesn't terminate.

Originally I had been matching on the error responses to get like the code, but then at some point the order changed for some reason and to keep from that hurting us again we use keyfind now everywhere after an error if we care what the error was.

A nice way to allow for pattern matching for this would be to return a map.

Would you be interested in such a change? I could put it behind a compile time configuration or runtime if you want to keep the default as a list of tuples.

I did a select of a table with a type like "column CHAR(40) NOT NULL" and the select returned a "{bpchar, Value}" tuple within the Rows list Row tuple, instead of just the Value. That is an error, right?

Try this:

pgsql_connection:simple_query("SELECT 1.0", {pgsql_connection, C}). %=> {{select,1},[{1.0}]}

pgsql_connection:extended_query("SELECT 1.0", [], {pgsql_connection, C}). %=> {{select,1},[{1}]}

I'm seeing a crash when postgres returns a timeout error when I run query with simple_query and I seem to have tracked it down to this anonymous function fun({set, []}) which would crash if the async return is an error and not that tuple.

I'm trying to understand the reasoning for these steps in pgsql_simple_query/4 and was hoping someone could explain what is going on in that functions.

I recently had some issues when starting a large number of connections quickly. I am not sure if I reached the maximum number of connection during a spike on my server but the pgsql_connection_sup gets stuck from time to time and its mailbox fill-up until I a manually restart the node or kill the process.

The supervisor seems to get stuck in here https://github.com/semiocast/pgsql/blob/master/src/pgsql_connection.erl#L453. Adding a short timeout like {timeout, 500} seems to do solve the problem.

It would be nice to have this setting as an option. I can try to work on a PR if you are interested.

iodata = iolist() | binary()

Did you plan to add JSON and JSONB support?
For now to insert or update I need use this ($1::text)::jsonb
And for select this field::json where I got raw json string as** {json, Json}** or raw jsonb as {jsonb, JsonB}

For example,

Decoding null to undefined when decoding from Postgres and encoding undefined to null.

Hi

Running a query using simple_query I got this Error:

{function_clause,[{pgsql_connection,set_succeeded_or_within_failed_transaction,[{error,closed}],[{file,"xxx/_build/default/lib/pgsql/src/pgsql_connection.erl"},{line,778}]},.....

Would adding {error, closed} clause solve the problem?

Is there a reason timestamps aren't returned in Erlang timestamp() format?

E.g.

iex(6)> Ecto.Pool.query "SELECT NOW()"
{1,[{{{2013,3,21},{18,35,0.241573}}}]}

I was testing using pgsql_connection:simple_query/4 with a sequence of SQL commands that create a transaction (BEGIN/(COMMIT|ROLLBACK)) and found I was able to crash the pgsql_connection pid at https://github.com/semiocast/pgsql/blob/master/src/pgsql_connection.erl#L699 due to the fun argument not handling "{error, {pgsql_error, [...proplist data...]}}" (i.e., instead it expects to always get "{set, []}"). This occurred when one of the SQL statements supplied was invalid. This should be an error that is not yet handled within pgsql_connection, since the Erlang pid should not be dying in this case.

Need a way to know what columns are associated with the row values.

Looks like some types are not being mapped before a fold:

1> pgsql_connection:simple_query("select prefix from prefixes_v6_unicast limit 1", C).
{{select,1},[{{cidr,<<"2403:c380::/32">>}}]}

2> pgsql_connection:fold(fun(A, Acc) -> io:format("A: ~p~n", [A]) end, null, "select prefix from prefixes_v6_unicast limit 1", C).
A: {{cidr,<<3,32,1,16,36,3,195,128,0,0,0,0,0,0,0,0,0,0,0,0>>}}
{ok,ok}

In the first case, for a cidr - you get a sane representation - but the second one is unusable in erlang (looks like the wire format?). Same for v4 cidr prefixes too.

For a pgsql_connection:extended_query of the form
INSERT INTO address (v, acct_id, org_id,)
VALUES ( $1, $2, $3)
RETURNING id;

The return value is e.g.
{{insert, 0, 1}, {[{record, <<00,08....>>}]}}
I.e. the return value is not decoded from the raw binary as it is for a SELECT query.

Using pgsql_connection:simple_query instead, gives e.g.
{{insert, 0, 1}, {[{21}]}}

It would be great to be able to use the set and return pattern as it eliminates one DB round trip.

This is an Elixir stacktrace, but it's readable:

iex(24)> Ecto.Pool.query "SELECT 0.0"
** (exit) {{:function_clause,[{:pgsql_protocol,:decode_numeric_bin_scale,[0,0],[file: 'src/pgsql_protocol.erl', line: 756]},{:pgsql_protocol,:decode_numeric_bin,1,[file: 'src/pgsql_protocol.erl', line: 743]},{:pgsql_protocol,:decode_row0,5,[file: 'src/pgsql_protocol.erl', line: 531]},{:pgsql_connection,:pgsql_extended_query_receive_loop,7,[file: 'src/pgsql_connection.erl', line: 824]}]},{:gen_server,:call,[#PID<0.167.0>,{:do_query,{:extended_query,"SELECT 0.0",[],[],:infinity}},:infinity]}}

I see hints in the source that it might be possible, but I can't see how to perform a COPY (like PQputCopyData and PQputCopyEnd in libpq). Is this possible?

support jsonb is in your plan?