alexeyr / erlang-sqlite3

Sqlite gen_server port for Erlang. Creates, reads and writes to sqlite database.

Erlang wrapper for SQLite3

This library allows you to work with SQLite3 databases from Erlang.

It is compatible with Windows and Linux, and should probably work on other OSes as well.

See also esqlite for an alternative library.

Requirements

Erlang/OTP R14B or later is required (tested up to 17.3 at this writing), and SQLite 3 minimum version is 3.6.1.

Compiling

Linux

1. Install SQLite3 by running sudo apt-get install sqlite3 or the equivalent for your package manager, or by compiling from the source.

2. make.

Cross-compiling

If you want to use erlang-sqlite3 on an embedded device, it can be cross-compiled.

1. Cross-compile SQLite3 and Erlang.

2. Change variables and paths in rebar.cross_compile.config.sample to the desired values and rename it to rebar.cross_compile.config.

3. make cross_compile.

Windows with MS Visual C++

To build both SQLite3 and sqlite3-erlang:

1. If MSVC tools (cl, link, etc.) are not in the path, run vcvars32.bat or vcvars64.bat depending on whether you use 32-bit or 64-bit Erlang. build_port_win32.bat and build_port_win64.bat have the standard paths for VC10.0.

2. nmake.

Alternately, you can use prebuilt versions of sqlite3.dll and sqlite3.def. To make sqlite3.lib, use lib /def:sqlite3.def. Then remove sqlite3.dll and sqlite3.lib targets from Makefile and do as above.

Potential compilation problems

• If SQLite was built with SQLITE_OMIT_LOAD_EXTENSION option, you'll need to undefine ERLANG_SQLITE3_LOAD_EXTENSION macro in <c_src/sqlite3_drv.h>.

Running the test suite

Linux

make test

Windows

1. nmake tests

2. If you get the error "Error loading sqlite3_drv: The specified module could not be found", this is because sqlite3.dll isn't in the search path.

Example usage

See tests test/sqlite3_test.erl for a starting point. On Windows note that sqlite3.dll must be in your application's working directory or somewhere in the DLL search path.

Authors

See ./AUTHORS

erlang-sqlite3's Issues

Create anonymous connection

Is it possible to add an option to create anonymous connection?
I need to create many temporary DBs over time and using atoms will force me to fill the atom table.
Maybe create a named sever only if DB is other than the atom undefined:
https://github.com/alexeyr/erlang-sqlite3/blob/master/src/sqlite3.erl#L127

case DB of
  undefined ->
    gen_server:start_link(?MODULE, Opts, []).
   _ ->
    gen_server:start_link({local, Db}, ?MODULE, Opts, [])
end.

Thanks

Error loading sqlite3_drv: The specified module could not be found

Hello,

I compile erlang-sqlite3. Thank i try make test and after this i get:

Error loading sqlite3_drv: The specified module could not be found

In README file write that: If you get the error "Error loading sqlite3_drv: The specified module could not be found", this is because sqlite3.dll isn't in the search path. Copy it to the .eunit directory.

If i under Linux? How can i fix this?

Thank you.

Store binary with latin1

User came with user_agent <<"^PUjC^PUjC°æ°">>

Sqlite driver failed trying to write it to database:

** exception exit: {{badarg,[{re,replace,
                                 [<<"^PUjC^PUjC°æ°">>,"'","''",[global,unicode]],
                                 [{file,"re.erl"},{line,357}]},
                             {sqlite3_lib,value_to_sql,1,
                                          [{file,"libs/sqlite3/src/sqlite3_lib.erl"},{line,107}]},
                             {sqlite3_lib,map_intersperse,3,
                                          [{file,"libs/sqlite3/src/sqlite3_lib.erl"},{line,311}]},

I don't know what is the right way to handle such input data? It is a text, but escape function is more targeted to a unicode list.

Maybe I should store all user input as a blob?

Or add a special case when I pass binary to use not re, but binary:replace?

"Can't write log file" --> segfault

LOGPATH defaults to /tmp/erlang-sqlite3-drv.log

If that file already exists via another user in a multi-user environment and we can't write to it we get "Can't write log file" https://github.com/alexeyr/erlang-sqlite3/blob/master/c_src/sqlite3_drv.c#L66

Some point after that we get a segfault, not sure where yet.

Removing the existing /tmp/erlang-sqlite3-drv.log file and re-running fixes temporarily but if 2 users are testing separately there is logfile contention.

We can change LOGPATH in every user's build but we really shouldn't segfault by default.

sqlite3_drv/get_columns is unsafe for scripts

pointer returned from sqlite3_column_name is not safe to use if prepare statement is called a second time. Invalid result can easily be reproduced:

sqlite3:sql_exec_script(Db,["SELECT * FROM __term;SELECT * FROM __term;"]).
[[{columns,[[192,20],[120,15,5,18]]},{rows,[]}],
[{columns,["id","term"]},{rows,[]}]]

Once dataset is sent back to erlang, pointers to column names of all but the last select will be invalid.

Null characters not allowed in strings

It would be nice to have all possible characters in strings. Approaches to investigate: http://bytes.com/topic/python/answers/26810-strings-null-bytes-inside-sqlite
http://www.ch-werner.de/javasqlite/SQLite/StringEncoder.html

error logging

I'm currently working with your library and I'm having some problems with the way you log/report errors. The actual line that annoys me a bit is https://github.com/alexeyr/erlang-sqlite3/blob/master/src/sqlite3.erl#L1038

I do not want the library to log errors for me. The library should return error tuples and I can handle the logging in my application code if I want to. Do you think we can change code? I can add a pull request to this ticket if you like. If there is another way to disable the logging please let me know!

Thanks for maintaining the library! Seems to be nice really nice so far.

Best Regards
Philipp

Move to rebar3-compatible build

At some point the build should use a rebar3-compatible Makefile to compile the C code: http://www.rebar3.org/docs/building-cc

Test fails on OS X

When running tests against master I get the following:

======================== EUnit ========================
module 'sqlite3_test'
  sqlite3_test: all_test_ (basic_functionality)...[0.002 s] ok
  sqlite3_test: all_test_ (table_info)...[0.001 s] ok
  sqlite3_test: all_test_ (parametrized)...[0.002 s] ok
  sqlite3_test: all_test_ (negative)...[0.001 s] ok
  sqlite3_test: all_test_ (blob)...[0.001 s] ok
  sqlite3_test: all_test_ (escaping)...[0.001 s] ok
  sqlite3_test: all_test_ (select_many_records)...[0.030 s] ok
  sqlite3_test: all_test_ (nonexistent_table_info)...[0.001 s] ok
  sqlite3_test: all_test_ (large_number)...[0.001 s] ok
  sqlite3_test: all_test_ (unicode)...[0.001 s] ok
  sqlite3_test: all_test_ (acc_string_encoding)...[0.001 s] ok
  sqlite3_test: all_test_ (large_offset)...[0.001 s] ok
  sqlite3_test: all_test_ (issue23)...[0.002 s] ok
  sqlite3_test: all_test_ (issue13)...[0.001 s] ok
  sqlite3_test: all_test_ (enable_load_extension)...ok
  sqlite3_test: all_test_ (changes)...[0.001 s] ok
  sqlite3_test: anonymous_test...Assertion failed: (0), function hash, file /SourceCache/cmph/cmph-1/src/hash.c, line 35.
make: *** [test] Abort trap: 6

I think this code block is causing it, which you can find around line 25352 in sqlite3.c in the amalgamation.

/* Remove all entries from a hash table.  Reclaim all memory.
** Call this routine to delete a hash table or to reset a hash table
** to the empty state.
*/
void sqlite3HashClear(Hash *pH){
  HashElem *elem;         /* For looping over all elements of the table */

  assert( pH!=0 );

Markdown EDoc

Review EDoc and add it to Wiki. Use https://github.com/esl/edown

128..255 < -1

When using sql_exec and binding Params the values 128..255 do not behave as expected; they compare < -1.
If these values are used inline in the query the results are as expected.
Example:

(dev@localhost)81> sqlite3:sql_exec(test128, "select foo from test where foo > 128;").
[{columns,["foo"]},{rows,[{255},{256}]}]
(dev@localhost)83> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [128]).
[{columns,["foo"]},
 {rows,[{-1},{0},{127},{128},{255},{256}]}]

128..255 must be being interpreted as chars? Float works ok...

(dev@localhost)87> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [128.0]).
[{columns,["foo"]},{rows,[{255},{256}]}]

How can I prevent this from happening with ints?

I was able to work around this issue via

([email protected])90> sqlite3:sql_exec(test128, "select foo from test where foo > :1;", [{1,128}]).
[{columns,["foo"]},{rows,[{255},{256}]}]

...but even looking at decode_and_bind_param in sqlite3_drv.c I'm not sure why 128 is being interpreted as it is. Any ideas?

Here's how we get here:

(dev@localhost)71> sqlite3:open(test128, [in_memory]).
{ok,<0.26992.1>}
(dev@localhost)72> sqlite3:sql_exec(test128, "create table test ( foo int )").
ok
(dev@localhost)73> sqlite3:sql_exec(test128, "insert into test values (-1)"). 
{rowid,1}
(dev@localhost)74> sqlite3:sql_exec(test128, "insert into test values (0)"). 
{rowid,2}
(dev@localhost)75> sqlite3:sql_exec(test128, "insert into test values (127)").
{rowid,3}
(dev@localhost)76> sqlite3:sql_exec(test128, "insert into test values (128)").
{rowid,4}
(dev@localhost)77> sqlite3:sql_exec(test128, "insert into test values (255)").
{rowid,5}
(dev@localhost)78> sqlite3:sql_exec(test128, "insert into test values (256)").
{rowid,6}
(dev@localhost)79> sqlite3:sql_exec(test128, "select foo from test;").                 
[{columns,["foo"]},
 {rows,[{-1},{0},{127},{128},{255},{256}]}]
(dev@localhost)80> sqlite3:sql_exec(test128, "select foo from test where foo > 127;").
[{columns,["foo"]},{rows,[{128},{255},{256}]}]
(dev@localhost)81> sqlite3:sql_exec(test128, "select foo from test where foo > 128;").
[{columns,["foo"]},{rows,[{255},{256}]}]
(dev@localhost)82> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [127]).
[{columns,["foo"]},{rows,[{128},{255},{256}]}]
(dev@localhost)83> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [128]).
[{columns,["foo"]},
 {rows,[{-1},{0},{127},{128},{255},{256}]}]
(dev@localhost)84> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [255]).
[{columns,["foo"]},{rows,[{0},{127},{128},{255},{256}]}]
(dev@localhost)85> sqlite3:sql_exec(test128, "select foo from test where foo > ?;", [256]).
[{columns,["foo"]},{rows,[]}]

memory leak for scripts

Last executed statement in a script does not get finalized in function sql_exec_async. This also means that if a script is called, the database handle never gets closed because there are unfinalized statements.

enable_load_extension

Is there a way to call the c function enable_load_extension from erlang?

https://www.sqlite.org/c3ref/enable_load_extension.html

Best Regards
Philipp

enable_load_extension test fails

Running sqlite3 version 3.7.9, and erl R14B04. After make to compile as per doc, make test fails with the following error message:

  sqlite3_test: all_test_ (enable_load_extension)...*failed*
::error:{assertEqual_failed,
          [{module,sqlite3_test},
           {line,340},
           {expression,"sqlite3 : enable_load_extension ( ct , 1 )"},
           {expected,ok},
           {value,
               {error,21,
                   "extension loading not enabled, recompile erlang-sqlite3 with ERLANG_SQLITE3_LOAD_EXTENSION defined"}}]}
  in function sqlite3_test:'-enable_load_extension/0-fun-0-'/1

The solution is to uncomment c_src/sqlite3_drv.h:7 - IMO all tests should pass in default install, is there a good reason the line is commented out?

sql_exec_script returns wrong answers

sql_exec_script("select * from user; update user set userid=1 where userid=2").
the update statement will corrupt the select results thus the returning values are wrong.

sqlite3:sql_exec crash on a select (some row with null column, others not)

With the table given below, sqlite3:sql_exec crashes.

I am using erlang-sqlite3 branch master 8752bff.

Steps to reproduce :

[{columns,["title","owner","status","date"]},
 {rows,[{<<"critical error should be simple error">>,1,1,
         null}]}]
4> sqlite3:sql_exec(db, "select \* from issue where rowid=2;").

Crash dump was written to: erl_crash.dump
temp_alloc: Cannot reallocate 2147483648 bytes of memory (of type "estack").
Aborted

Database used:

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE issue (title TEXT, owner INT, status INT, date int);
INSERT INTO "issue" VALUES('critical error should be simple error',1,1,NULL);
INSERT INTO "issue" VALUES('typo in comment',1,1,1349379571);
INSERT INTO "issue" VALUES('Serious slow-down at startup',1,1,NULL);
INSERT INTO "issue" VALUES('empty plc database',1,1,NULL);
COMMIT;

>64-bit 'offset' value blows up

sqlite3 handles >64-bit offset...

$ sqlite3
sqlite> create table foo(id int);
sqlite> select * from foo limit 1 offset 9223372036854775807;
sqlite> select * from foo limit 1 offset 9223372036854775808;
Error: datatype mismatch

erlang-sqlite3 fails badly on it.

Statically link driver with sqlite3

I need to statically link this driver - I'm going to start on this but I'm wondering if anyone's tried this and if there are any known issues.

Symbol not found: _sqlite3_enable_load_extension

e417a95 is broken for me, probably because i don't have sqlite3 extension loading compiled in

is there a way to test whether the feature is available in sqlite3, or a macro to use? otherwise any optional extras you add will need their own #ifdef...#endif

from http://www.sqlite.org/compile.html#omit_load_extension ...

The -ldl library is needed to support dynamic loading, the sqlite3_load_extension() interface and the load_extension() SQL function. If these features are not required, then they can be omitted using SQLITE_OMIT_LOAD_EXTENSION compile-time option:

gcc -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION shell.c sqlite3.c

$ less crash.log
2013-01-03 14:54:21 =CRASH REPORT====
crasher:
initial call: sqlite3:init/1
pid: <0.125.0>
registered_name: []
exception exit: {"Error loading sqlite3_drv: dlopen(deps/sqlite3/priv/sqlite3_drv.so, 2): Symbol not found: _sqlite3_enable_load_extension\n Referenced from: /Users/rflynn.../deps/sqlite3/priv/sqlite3_drv.so\n Expected in: flat namespace\n in /Users/rflynn.../deps/sqlite3/priv/sqlite3_drv.so

Large integer numbers don't work

This test fails.

large_number_test() ->
    sqlite3:open(ct),
    N1 = 4294967295,
    N2 = (N1 + 1) div 2,
    Query1 = io_lib:format("select ~p, ~p", [N1, N2]),
    ?assertEqual([{N1, N2}], rows(sqlite3:sql_exec(ct, Query1))).

Windows: Error loading sqlite3_drv: The specified module could not be found.

I'm having the same problem under windows. The get_priv_dir/0 gives the path to the priv dir but the I'm not able to load the dll.

1> PD = sqlite3:get_priv_dir().
"d:/Dev/erl/erlang-sqlite3/ebin/../priv"
2> erl_ddll:load(PD, "sqlite3_drv").
{error,{open_error,-136}}

Similar issue at #4.

Windows XP Pro 32-bit
Erlang/OTP 17 erts-6.0
Visual Studio 10.0
Sqlite 3.7.11

When the emulator is launched with async threads, sql_exec_script doesn't always work correctly

The reason is that effectively, all statements are first prepared and then executed. If one statement creates a table, and another creates an index for that table, the second one won't be prepared because the table doesn't exist yet.

It seems the only possibility is to expand async_sqlite3_command. Probably it needs to be made into a union (one option for statements, other option for scripts, some more?)

Safe to execute arbitrary string?

I want to execute untrusted user submitted string with sql_exec/1 and return the result.
It's OK that the user will read all data and even delete the DB.

The SQLite shell let the user access the filesystem with ".read FILENAME"
In my case a user might try to read arbitrary file and get the content as the shell error.
Does erlang-sqlite3 allow to pass only valid SQL to sql_exec/1 so special commands like ".read FILENAME" won't work and the user will only be able to access the DB?

Thanks

Provide easy to use buffer for writes

SQLite does not support concurrent writes. It would be great if erlang-sqlite3 provided an easy-to-use queue that INSERT commands could be sent to via multiple processes, but that would be written sequentially to the database...