staab / janet-pqutils Goto Github PK
View Code? Open in Web Editor NEWA utility library for Janet and janet-pq.
License: MIT License
janet-pqutils's People
Stargazers
Watchers
janet-pqutils's Issues
Add support for cursors
The current "lazy" implementation only lazily grabs results from libpq's result object which lives in memory, not from the server, so it doesn't actually help anything. In order to do that, we need to use cursors.
Clean up oid matching
Matching on strings is a performance problem since it's in an inner loop. We should convert this to some constant-time match functionality in C for custom types, and for built-in types bring in pg_types.c. Once we do that, #7 may just go away.
Fix segfault
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==11071==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x7f94ecbfb240 (pc 0x7fff633066f2 bp 0x7ffeea954ae0 sp 0x7ffeea954ae0 T7799094)
==11071==The signal is caused by a READ memory access.
#0 0x7fff633066f1 in _platform_strlen (libsystem_platform.dylib:x86_64+0x16f1)
#1 0x1052d2c02 in janet_cstring (janet:x86_64+0x100028c02)
#2 0x10532c1b3 in populate_oids (core.so:x86_64+0x11b3)
#3 0x10532c74f in cfun_exec (core.so:x86_64+0x174f)
#4 0x1052dcea3 in run_vm (janet:x86_64+0x100032ea3)
#5 0x1052ddbdd in janet_continue (janet:x86_64+0x100033bdd)
#6 0x1052dd14e in run_vm (janet:x86_64+0x10003314e)
#7 0x1052ddbdd in janet_continue (janet:x86_64+0x100033bdd)
#8 0x1052cf981 in janet_dobytes (janet:x86_64+0x100025981)
#9 0x1052df7f4 in main (janet:x86_64+0x1000357f4)
#10 0x7fff6311e3d4 in start (libdyld.dylib:x86_64+0x163d4)
==11071==Register values:
rax = 0x415ad6da00000000 rbx = 0x415ad6da00000000 rcx = 0x00007f94ecbfb240 rdx = 0x00007f94ecbfb240
rdi = 0x00007f94ecbfb240 rsi = 0x00007f94ec400000 rbp = 0x00007ffeea954ae0 rsp = 0x00007ffeea954ae0
r8 = 0x0000000000000006 r9 = 0x0000000000000000 r10 = 0x0000000000000007 r11 = 0x0000000000000007
r12 = 0x00007f94ec634ec0 r13 = 0x0000000000000011 r14 = 0x00007f94ecbfb240 r15 = 0x00007f94ec502d01
UndefinedBehaviorSanitizer can not provide additional info.
==11071==ABORTING
Add extensible result unpacking
For stuff not handled directly in core.c, we should add a multimethod that can be populated with user-defined functionality for unpacking rows. We should run all results through an unpack function by default, but allow turning it off with a dynamic binding.
Switch away from gensym for cursor names
Using e.g., (base16/encode (os/cryptorand 16))
Add transaction helper
With different nesting modes:
- :noop - nested transactions just continue the current transaction if there is one.
- :savepoint - create a savepoint to roll back to
- :disallow - throw on error on nested transaction. Probably the default.
Add exec_lazy
As far as I can tell, the most general purpose way to return results would be to create a fiber and yield from within it. This would require creating a result wrapper that would call PQclear when garbage collected. It would also require figuring out how the heck to yield from a c function 🤔.
Add query dsl
Three layers:
- Something data-oriented like
{:select [:x :y] :where [{:x := 1}]}, which allows us to target clauses by type when compiling the query. This might be redundant and unnecessary given representation number 2. - Also data-oriented, but as a collection, like
[{:t :select :v :x} {:t :select :v :y} {:t :where :k :x :op := :v 1}]. This allows the user to easily compose/modify queries, since a query is a (recursive, with subqueries) collection of clauses. - A convenience layer, e.g.
(q (select x y z) (where= x 1)). Theqfunction should compile directly to the collection representation above, flattening values returned from query building functions/macros into a single list. This should recur into subqueries as well.
A compile function should take any of the three representations above and turn it into a string, escaping literals and identifiers as necessary. If the input is a string already, it should just return it.
Execution functions shouldn't know about this layer, but should expose a configuration option for preparing queries, which can be configured by setting it to compile.
Check error when catching in iter
We catch an error for retrieving next rows, but we should check the message.
Use s64 or u64 for types that won't fit into a float.
I think when reading a serial or big serial for example, it might be better to use s64 than a plain janet number for correctness sake.
Get the thing compiling
For additional background on getting and installing libpq, check out this SO answer.
This is just a dumping place for my progress in compiling/linking my code with postgres. Unless otherwise specified, I'm using jpm test to trigger the errors.
After getting the right -I flag to include libpq-fe.h, I started getting:
creating static library build/pg.a...
running test/pg.janet ...
error: could not load native build/pg.so: dlopen(build/pg.so, 2): Symbol not found: _PQconnectdb
Referenced from: build/pg.so
Expected in: flat namespace
in build/pg.so
in native
in <anonymous> [boot.janet] on line 1934, column 26
in require [boot.janet] on line 1955, column 16
in import* [boot.janet] on line 1967, column 15
in _thunk [test/pg.janet] (tailcall) on line 1, column 1
I was able to get that error to go away by including all the c files from libpq:
(declare-native
:name "pg"
:source @["staab.pg/pg.c"
"postgresql/src/interfaces/libpq/fe-auth-scram.c"
"postgresql/src/interfaces/libpq/fe-auth.c"
"postgresql/src/interfaces/libpq/fe-connect.c"
"postgresql/src/interfaces/libpq/fe-exec.c"
"postgresql/src/interfaces/libpq/fe-gssapi-common.c"
"postgresql/src/interfaces/libpq/fe-lobj.c"
"postgresql/src/interfaces/libpq/fe-misc.c"
"postgresql/src/interfaces/libpq/fe-print.c"
"postgresql/src/interfaces/libpq/fe-protocol2.c"
"postgresql/src/interfaces/libpq/fe-protocol3.c"
"postgresql/src/interfaces/libpq/fe-secure-common.c"
"postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
"postgresql/src/interfaces/libpq/fe-secure-openssl.c"
"postgresql/src/interfaces/libpq/fe-secure.c"
"postgresql/src/interfaces/libpq/legacy-pqsignal.c"
"postgresql/src/interfaces/libpq/libpq-events.c"
"postgresql/src/interfaces/libpq/pqexpbuffer.c"
"postgresql/src/interfaces/libpq/pthread-win32.c"
"postgresql/src/interfaces/libpq/win32.c"]
:cflags @["-Ipostgresql/src/interfaces/libpq"
"-Ipostgresql/src/include"])
I then got:
compiling build/postgresql___src___interfaces___libpq___fe-connect.o...
postgresql/src/interfaces/libpq/fe-connect.c:32:10: fatal error: 'pg_config_paths.h' file not found
#include "pg_config_paths.h"
^~~~~~~~~~~~~~~~~~~
1 error generated.
Which I was able to get rid of by running cd postgresql/src/interfaces/libpq && make ../../../src/port/pg_config_paths.h and adding -Ipostgresql/src/port to my :cflags section.
After looking into it a bit more, that seemed like a bandaid, so I ended up installing the client libraries to postgresql/build, using:
cd postgresql
rm -rf build
make distclean
./configure --prefix=$(pwd)/build
make -C src/bin install
make -C src/include install
make -C src/interfaces install
make -C doc install
This let me to change my project.janet to:
(declare-native
:name "pg"
:source @["staab.pg/pg.c"
"postgresql/src/interfaces/libpq/fe-auth-scram.c"
"postgresql/src/interfaces/libpq/fe-auth.c"
"postgresql/src/interfaces/libpq/fe-connect.c"
"postgresql/src/interfaces/libpq/fe-exec.c"
"postgresql/src/interfaces/libpq/fe-gssapi-common.c"
"postgresql/src/interfaces/libpq/fe-lobj.c"
"postgresql/src/interfaces/libpq/fe-misc.c"
"postgresql/src/interfaces/libpq/fe-print.c"
"postgresql/src/interfaces/libpq/fe-protocol2.c"
"postgresql/src/interfaces/libpq/fe-protocol3.c"
"postgresql/src/interfaces/libpq/fe-secure-common.c"
"postgresql/src/interfaces/libpq/fe-secure-gssapi.c"
"postgresql/src/interfaces/libpq/fe-secure-openssl.c"
"postgresql/src/interfaces/libpq/fe-secure.c"
"postgresql/src/interfaces/libpq/legacy-pqsignal.c"
"postgresql/src/interfaces/libpq/libpq-events.c"
"postgresql/src/interfaces/libpq/pqexpbuffer.c"
"postgresql/src/interfaces/libpq/pthread-win32.c"
"postgresql/src/interfaces/libpq/win32.c"]
:cflags @["-Ipostgresql/build/include"
"-Ipostgresql/build/include/libpq"
"-Ipostgresql/build/include/internal"
"-Ipostgresql/build/include/internal/libpq"
"-Ipostgresql/build/include/server"
"-Lpostgresql/build/lib"])
At this point, I'm now getting a bunch of errors having to do with types that aren't defined:
compiling build/postgresql___src___interfaces___libpq___fe-gssapi-common.o...
clang: warning: argument unused during compilation: '-Lpostgresql/build/lib' [-Wunused-command-line-argument]
postgresql/src/interfaces/libpq/fe-gssapi-common.c:26:6: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 stat, int type)
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:28:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 lmin_s;
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:29:2: error: use of undeclared identifier 'gss_buffer_desc'
gss_buffer_desc lmsg;
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:30:2: error: unknown type name 'OM_uint32'; did you mean 'uint32'?
OM_uint32 msg_ctx = 0;
^~~~~~~~~
uint32
postgresql/build/include/internal/c.h:359:22: note: 'uint32' declared here
typedef unsigned int uint32; /* == 32 bits */
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:34:3: warning: implicit declaration of function
'gss_display_status' is invalid in C99 [-Wimplicit-function-declaration]
gss_display_status(&lmin_s, stat, type,
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:10: error: use of undeclared identifier 'GSS_C_NO_OID'
GSS_C_NO_OID, &msg_ctx, &lmsg);
^
postgresql/src/interfaces/libpq/fe-gssapi-common.c:35:35: error: use of undeclared identifier 'lmsg'
GSS_C_NO_OID, &msg_ctx, &lmsg);
(several more lines)
If I search postgresql for e.g., OM_uint32, I get a lot of uses of it, but no definition, so I wonder if it's a preprocessor thing. Maybe I need to reference different c files, or use a -L option?
In summary, I have no idea what I'm doing. I'll keep this updated as I work on it.
Support binary data
Both when collecting results and when passing parameters (see #3 for that).
Add support for parameters
We're already using PQexecParams, just gotta pass params through.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.