denodrivers / sqlite3

The fastest and correct SQLite3 module for Deno runtime

Deno SQLite3

The fastest and correct module for SQLite3 in Deno.

Example

import { Database } from "jsr:@db/[email protected]";

const db = new Database("test.db");

const [version] = db.prepare("select sqlite_version()").value<[string]>()!;
console.log(version);

db.close();

Usage

Since this library depends on the unstable FFI API, you must pass --allow-env, --allow-ffi and --unstable-ffi flags. Network and FS permissions are also needed to download and cache prebuilt library.

You can also just use --allow-all / -A flag since FFI basically gives full access.

deno run -A --unstable-ffi <file>

Benchmark

Benchmark based on just-js/02-sqlite

See bench for benchmarks source.

Documentation

See doc.md for documentation.

Check out the complete API reference here.

Native Library

It will download and cache a prebuilt shared library from GitHub releases, for which it will need network and file system read/write permission.

If you want to use custom library, then you can set the DENO_SQLITE_PATH environment variable, to a fully specified path to the SQLite3 shared library.

Contributing

Code is formatted using deno fmt and linted using deno lint. Please make sure to run these commands before committing.

You can optionally build sqlite3 from source. Make sure that you have the submodule (git submodule update --init --recursive).

deno task build

When running tests and benchmarks, you use the DENO_SQLITE_LOCAL=1 env variable otherwise it won't use to locally compiled SQLite library.

DENO_SQLITE_LOCAL=1 deno task bench

Related

• x/sqlite, WASM based.

License

Apache-2.0. Check LICENSE for details.

Copyright © 2023 DjDeveloperr

sqlite3's Issues

Error: Uncaught TypeError: Invalid ffi

Hi,

I'd like to use this library over the WASM based one so I can use WAL mode. Unfortunately the example code already fails for me with the error:

error: Uncaught TypeError: Invalid ffi arg value, expected TypedArray, UnsafePointer or null
  const result = lib.sqlite3_prepare_v2(
                     ^
    at prepareArgs (deno:ext/ffi/00_ffi.js:170:17)
    at Object.fn [as sqlite3_prepare_v2] (deno:ext/ffi/00_ffi.js:302:43)
    at sqlite3_prepare_v2 (https://deno.land/x/[email protected]/src/ffi.ts:499:22)
    at Database.prepare (https://deno.land/x/[email protected]/src/database.ts:222:20)
    at Database.queryArray (https://deno.land/x/[email protected]/src/database.ts:267:23)
    at file:///home/<snipped>/use-deno-driver.ts:5:22

it does create an empty test.db file. I use the so file from libsqlite3-dev, but I tried another one and had the same problem.

I use this to execute:

[12:53] bart@host:~$ uname -a
Linux host 5.10.0-17-amd64 #1 SMP Debian 5.10.136-1 (2022-08-13) x86_64 GNU/Linux
[12:53] bart@host:~$ deno --version
deno 1.23.0 (release, x86_64-unknown-linux-gnu)
v8 10.4.132.5
typescript 4.7.2
[12:53] bart@host:~$ deno run -A --allow-ffi --unstable use-deno-driver.ts 

What could be the cause of this error and what am I doing wrong?

example code:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("test.db");

const [version] = db.queryArray("select sqlite_version()")[0];
console.log(version);

db.close();

aliases?

could we add two aliases? query() for queryArray and queryEntries() for queryObject so its compatible with x/sqlite lib?

Get one row as object

Current .get returns an array, and you have to map keys and values again to create object.

Or something like that:

cosnt record = db
      .prepare("SELECT id, email, challenge FROM user WHERE id = ?")
      .all(user.id) as unknown as User[];
return record.length ? record[0] : null;

Maybe I have missed something in docs.

not working inside docker

Dockerfile

FROM denoland/deno:1.26.2
WORKDIR /app
ADD . .
RUN deno cache main.ts
CMD ["run", "-A", "--unstable", "main.ts"]

main.ts

import * as sqlite from "https://deno.land/x/[email protected]/mod.ts";

const db = new sqlite.Database("test.db");

const [version] = db.prepare("select sqlite_version()").value<[string]>()!;
console.log(version);

db.close();

docker build and run (failed silently without any error)

Namespace 'Deno' has no exported member 'ForeignFunction'

Hello. Running deno test results in many errors, such as:

error: TS2694 [ERROR]: Namespace 'Deno' has no exported member 'ForeignFunction'.

I am using the latest version of Deno today, version 1.19.0

deno 1.19.0 (release, x86_64-unknown-linux-gnu)
v8 9.9.115.7
typescript 4.5.2

I'm somewhat new to Deno... What am I missing?

NOTE: Before I was even able to run it, I had to modify test/test.ts because Deno was complaining that t.step doesn't exist. So, I created a mock version of that function to make it work.

Insertions in prepared statements need to be finalized to persist

Hi, just noticed that insertions in prepared statements won't persist to disk until the prepared statement is finalized.

Here's a snippet to reproduce the behavior:

// main.ts
import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const REPETITIONS = 5;

const db = new Database("test.db");
db.exec(
  "CREATE TABLE IF NOT EXISTS people (id INTEGER PRIMARY KEY, name STRING)",
);

for (let i = 1; i <= REPETITIONS; i++) {
  const stmt = db.prepare(
    "INSERT INTO people (name) VALUES (?) RETURNING id, name",
  );
  const [id, name] = stmt.value<[number, string]>("Johnny " + i)!;
  // stmt.finalize()
  console.log("Inserted:", id, name);
}

db.close();

deno run -A --unstable main.ts

Running multiple times the command above, will produce the same output:

$ deno run -A --unstable main.ts
Inserted: 1 Johnny 1
Inserted: 2 Johnny 2
Inserted: 3 Johnny 3
Inserted: 4 Johnny 4
Inserted: 5 Johnny 5
$ deno run -A --unstable main.ts
Inserted: 1 Johnny 1
Inserted: 2 Johnny 2
Inserted: 3 Johnny 3
Inserted: 4 Johnny 4
Inserted: 5 Johnny 5

When uncommenting the // stmt.finalize() line, insertions actually get persisted between executions:

$ deno run -A --unstable main.ts
Inserted: 1 Johnny 1
Inserted: 2 Johnny 2
Inserted: 3 Johnny 3
Inserted: 4 Johnny 4
Inserted: 5 Johnny 5
$ deno run -A --unstable main.ts
Inserted: 6 Johnny 1
Inserted: 7 Johnny 2
Inserted: 8 Johnny 3
Inserted: 9 Johnny 4
Inserted: 10 Johnny 5

I'm not sure if this is the correct behavior, but if it is, maybe it should be something explicit on the documentation for prepared statements?.

Thank you very much in advance :)

Segmentation fault (core dumped)

That's it. Same code as in #2

Bug in Reading Integers

I was trying to store timestamps as INTEGER values in SQLite3, but when reading the values it looks like they are messed up.

Filtering for the real integer value works (result contains the expected rows). Also when accessing the database via sqlite3 CLI everything looks okay. Therefore I guess the problem is somewhere in this module.

Please see the screenshot for an example:

Here's the script showing the problem:

import { Database as SQLite3 } from "https://deno.land/x/[email protected]/mod.ts"

const db = new SQLite3("test.db")

db.exec(`CREATE TABLE IF NOT EXISTS test (
  id TEXT PRIMARY KEY NOT NULL,
  created_at INTEGER NOT NULL
)`)

db.exec(`DELETE FROM test WHERE id = 'test-row' OR id = 'direct';`)

// insert some sample rows

const createdAt = (new Date('2023-03-29T13:04:00+02:00')).valueOf()
console.log({ createdAt })

db.prepare(`INSERT INTO test (id, created_at) VALUES (:id, :createdAt)`).run({
  id: 'test-row',
  createdAt,
})

db.prepare(`INSERT INTO test (id, created_at) VALUES ('direct', 1680087840000)`).run()

// get all rows
const rows = db.prepare(`SELECT * FROM test;`).all()

// log
console.table(rows)
console.table(rows.map(r => ({ id: r.id, createdAt: new Date(r.createdAt) })))

// proofing that SQLite has stored the correct values:
// filtering by exact timestamp
console.log(`Result of:\n   SELECT * FROM test WHERE created_at = 1680087840000;`)
console.table(db.prepare(`SELECT * FROM test WHERE created_at = 1680087840000;`).all())

Error: Native SQLite3 library not found, try installing SQLite3

After upgrade to 0.6.0 from 0.5.2 (OSX 12.6)

Download https://github.com/denodrivers/sqlite3/releases/download/v0.6.0/libsqlite3_aarch64.dylib
error: Uncaught (in promise) Error: Native SQLite3 library not found, try installing SQLite3. If you have an existing installation, either add it to path or set the `DENO_SQLITE_PATH` environment variable.
  const error = new Error(
                ^
    at https://deno.land/x/[email protected]/src/ffi.ts:450:17
Caused by: CacheError: Not Found
    at protocolHttp (https://deno.land/x/[email protected]/file_fetcher.ts:23:11)
    at async fetchFile (https://deno.land/x/[email protected]/file_fetcher.ts:45:14)
    at async FileWrapper.fetch (https://deno.land/x/[email protected]/file.ts:95:18)
    at async FileWrapper.get (https://deno.land/x/[email protected]/file.ts:113:12)
    at async cache (https://deno.land/x/[email protected]/cache.ts:67:10)
    at async Wrapper.cache (https://deno.land/x/[email protected]/cache.ts:21:12)
    at async download (https://deno.land/x/[email protected]/plug.ts:107:16)
    at async prepare (https://deno.land/x/[email protected]/plug.ts:116:16)
    at async https://deno.land/x/[email protected]/src/ffi.ts:434:12

I installed it with brew:

which sqlite3
/usr/bin/sqlite3

And added it to profiles:

echo 'export PATH="/opt/homebrew/opt/sqlite/bin:$PATH"' >> ~/.zshrc

But error is the same. 0.5.2 worked ok without even sqlite3 installed with brew.

Please, can someone give a hint how to fix it.

Does this solve the disadvantages of the x/sqlite library for Deno?

Disadvantages

• Speed: file system IO through Deno can be significantly lower compared to what is achievable using a native binary
• Weaker Persistence Guarantees: due to limitations in Denos file system APIs, SQLite can't acquire file locks or memory map files, which makes some persistence guarantees less strong (e.g. this module can't safely use WAL mode)

https://deno.land/x/[email protected]#comparison-to-plugin-based-modules

malloc_consolidate(): invalid chunk size

malloc_consolidate(): invalid chunk size
Aborted (core dumped)

It happens quite frequently.

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

db.execute("CREATE TABLE IF NOT EXISTS users (name TEXT NOT NULL)");

for (let i = 0; i < 10_000; i++) {
  db.execute("INSERT INTO users VALUES (?)", "littledivy");
}

Feature request: object parameters

Currently, parameters for queries use arrays only. Having objects for parameters would be great as they'd remove the need for parameters into carefully considered order and eliminates the need for repetition in queries which use the same parameter more than once.
Before

const rows = db.queryObject(`
  SELECT *
  FROM users
  WHERE role = ?
  LIMIT ?
  OFFSET (? * (? - 1));
`, role, postsPerPage, postsPerPage, page);

After

const rows = db.queryObject(`
  SELECT *
  FROM users
  WHERE role = ?
  LIMIT :postsPerPage
  OFFSET (:postsPerPage * (:page - 1));
`, { role, postsPerPage, page });

Transactions not working correctly

Currently, transactions are not working when using either named parameters or un-named parameters.
Setup:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database(":memory:");

db.execute(`
  CREATE TABLE people (
    name TEXT NOT NULL
  );
`);

Named parameters:

db.execute(`
  BEGIN TRANSACTION;
  INSERT INTO people (name)
  VALUES (:name);
  COMMIT;
`, { name: "John Doe" });

Throws:

error: Uncaught Error: Couldn't find index for ':name'
      throw new Error(`Couldn't find index for '${name}'`);
            ^
    at PreparedStatement.bindParameterIndex (https://deno.land/x/[email protected]/src/database.ts:386:13)
    at PreparedStatement.bindAllNamed (https://deno.land/x/[email protected]/src/database.ts:521:26)
    at Database.execute (https://deno.land/x/[email protected]/src/database.ts:148:14)

Un-named parameters:

db.execute(`
  BEGIN TRANSACTION;
  INSERT INTO people (name)
  VALUES (?);
  COMMIT;
`, "John Doe");

Throws:

error: Uncaught Error: (25) column index out of range: column index out of range
    throw new Error(`(${result}) ${sqlite3_errstr(result)}: ${msg}`);
          ^
    at unwrap_error (https://deno.land/x/[email protected]/src/ffi.ts:368:11)
    at sqlite3_bind_text (https://deno.land/x/[email protected]/src/ffi.ts:448:3)
    at PreparedStatement.bind (https://deno.land/x/[email protected]/src/database.ts:479:9)
    at PreparedStatement.bindAll (https://deno.land/x/[email protected]/src/database.ts:515:12)
    at Database.execute (https://deno.land/x/[email protected]/src/database.ts:150:14)

Deno version: 1.17.3 (release, x86_64-apple-darwin)
Module version: 0.3.0
macOS version: 12.1
SQLite version: 3.37.2

[uncaught application error]: TypeError - db.execute is not a function

Getting this error now on 0.5.0

Doesn't work on Linux

$ deno run -A ./app.ts  
Download https://cdn.skypack.dev/error/buffer?from=vfile
error: Uncaught Error: Native SQLite3 library not found, try installing SQLite3. If you have an existing installation, either add it to path or set the `DENO_SQLITE_PATH` environment variable.
  const error = new Error(
                ^
    at https://deno.land/x/[email protected]/src/ffi.ts:426:17
Caused by: TypeError: Deno.dlopen is not a function
    at https://deno.land/x/[email protected]/src/ffi.ts:420:14
➜  app git:(master) ✗ 
$ deno -V             
deno 1.25.0

Manjaro/arch and Ubuntu. Neither work all of a sudden.

How to run arbitrary statements?

await db.queryObject("PRAGMA busy_timeout = 30000");

This doesn't work. How would I run some random SQL statements?

error: Uncaught SyntaxError: Unexpected reserved word

db.queryArray does same thing.

Error: Native SQLite3 library not found, try installing SQLite3.

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

// Open a database
export const client = new Database("../tolol.db");

info :
deno --version
deno 1.23.3 (release, x86_64-unknown-linux-gnu)
v8 10.4.132.8
typescript 4.7.4

error message :

error: Error: Native SQLite3 library not found, try installing SQLite3. If you have an existing installation, either add it to path or set the DENO_SQLITE_PATH environment variable.
at https://deno.land/x/[email protected]/src/ffi.ts:425:17

Issue with Native libsqlite3.so

That may be mine issue*

• Your code works just fine for windows but I cannot make it working on my ubuntu server. I've installed sqlite3 itself & libsqlite3-0. Library is properly added into bash path, I even manually added DENO_SQLITE_PATH that points to "/lib/x86_64-linux-gnu/libsqlite3.so.0" (default installation path). Still, I got:

May 16 19:48:37 lightcluster deno[2207]: error: Uncaught Error: Native SQLite3 library not found, try installing SQLite3. If you have an existing installation, either add it to path or set the `DENO_SQLITE_PATH` environment variable.
May 16 19:48:37 lightcluster deno[2207]:     const error = new Error(
May 16 19:48:37 lightcluster deno[2207]:                   ^
May 16 19:48:37 lightcluster deno[2207]:     at https://deno.land/x/[email protected]/src/ffi.ts:431:19
May 16 19:48:37 lightcluster deno[2207]: Caused by: Uncaught Error: Could not open library: Could not open library: libsqlite3.so: cannot open shared object file: No such file or directory
May 16 19:48:37 lightcluster deno[2207]:     lib = Deno.dlopen(
May 16 19:48:37 lightcluster deno[2207]:                ^
May 16 19:48:37 lightcluster deno[2207]:     at Object.opSync (deno:core/01_core.js:177:12)
May 16 19:48:37 lightcluster deno[2207]:     at new DynamicLibrary (deno:ext/ffi/00_ffi.js:231:24)
May 16 19:48:37 lightcluster deno[2207]:     at Object.dlopen (deno:ext/ffi/00_ffi.js:329:12)
May 16 19:48:37 lightcluster deno[2207]:     at https://deno.land/x/[email protected]/src/ffi.ts:418:16

I'm defeated, all other apps that uses sqlite just works normally.

Fts5

Hi, I am looking for an sqlite module with fts support. This module looks pretty good, but I could not find any information about fts. As far as I know fts is a standard feature from sqlite which you have to enable threw command arguments. How do I enable the extension in this case?

Can't Deno.removeSync folder containing Sqlite db cleanly until after lstatSync call.

This is probably not a bug for this project, but just more of a note for anyone that might be using an Sqlite db in a temp folder...

I have just converted all of my program's shutdown behavior to be synchronous for reasons discussed here.

When I converted code using an Sqlite database, I ran into an issue where calling Deno.removeSync(workPath, { recursive: true }); was raising a NotFound error for the temp folder (workPath below) even though the folder was eventually being deleted...

Previously when I was using Deno.remove asynchronously I wasn't having this problem even though I was awaiting all of the async calls before Deno.exit eventually got called, after catching SIGINT....

Anyway, here was the solution for me: Just call Deno.lstatSync once before calling Deno.removeSync on the directory... With this in place, there are no errors during shutdown and my program also exits before the Deno watcher does.

close() {
    try {
      const { db } = this;
      db.close();
    } catch (ex) {
      console.log("db.close", ex);
    }
    // #region ATTENTION
    //
    // Do one synchronous existence check for workPath after closing the db and
    // before trying to delete the folder by calling `Deno.removeSync`. This
    // avoids an incorrect "NotFound" error during that call.
    //
    // Could be caused by Sqlite still having a file locked? Not sure...
    //
    // #endregion
    try {
      Deno.lstatSync(this.workPath);
    } catch (err) {
      if (err instanceof Deno.errors.NotFound) {
        return;
      }
      console.log("Workpath access error", err);
    }
    try {
      Deno.removeSync(this.workPath, { recursive: true });
    } catch (ex) {
      console.log("Removing work path", ex);
    }
  }

Bundle SQLite3 dynamic library

Why does this library not come with a bundled version of the SQLite3 dynamic library?
Especially in a system like deno, where the versions and locations of all other dependencies are specified as url-s in the code, having to manually install and manage the version of a SQLite3 dynamic library feels clumsy and error prone.

Support extensions

I am looking at using SqLite with SpatiaLite but that would require the implementation of the load_extension api.

https://www.sqlite.org/c3ref/load_extension.html

[uncaught application error]: TypeError - db.queryObject is not a function

Getting this on 0.5.0 now

[Deno deploy] Native SQLite3 library not found

Try to run this code on Deno deploy:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("test.db");

const [version] = db.queryArray("select sqlite_version()")[0];
console.log(version);

db.close();

https://dash.deno.com/playground/heavy-mouse-20

and got this error:

Error: Native SQLite3 library not found, try installing SQLite3. If you have an existing installation, either add it to path or set the `DENO_SQLITE_PATH` environment variable.
    at https://deno.land/x/[email protected]/src/ffi.ts:401:23

No RestBindParameters for 'Statment.get'

First of all, congratulations on the latest release I'm really liking the new API and functions, it's so clean.

The only problem I have with it, is that according to the doc.md, the Statment.get() could be given some params, same as all and values, but this doesn't seems to be the case. Is there a reason for this? Or maybe I'm seeing it wrong and this is the expected behavior and didn't understand how get works.

Thank you for your time.

Query parameters

Hi there, I'm wanting to make queries with dynamic parameters but cannot find any documentation regarding it. Is this simply something that hasn't been done yet?

PS. So far, this is looking like a very quality library. I'm very keen to see what it evolves into :)

Ask for the required flags instead of showing a generic message

Hi, when executing the example in the README.md with:

deno run --unstable --allow-env mod.ts

Throws saying it cannot find the SQLite library:

But executing it with the --allow-all it works pretty good:

I think it should ask for the allow-ffi flag in order to execute it right:

Thank you!

[uncaught application error]: TypeError - Invalid CString pointer, not valid UTF-8

Getting this error all of a sudden.

I wonder if maybe some non-utf8 strings got into my db? I donj't know how to fix.

Here's the query:

  .get('/recent', async (context) => {
    const links = await context.state.db
      .prepare(
        'SELECT url, title, excerpt, created_at, count FROM urls ORDER BY created_at DESC LIMIT 50'
      )
      .all();

    context.response.body = links;
  })

Breaks with Deno v1.30

When upgrading from Deno 1.29 to 1.30, I encountered the following error:

error: Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'ops')
const { op_ffi_cstr_read, op_ffi_get_buf } = (Deno as any).core.ops;
                                                                ^
    at https://deno.land/x/[email protected]/src/util.ts:19:65

As a temporary workaround I reverted back 1.29.

Unsafe pointer create exception

Hi,
i tried to use the newest version of sqlite3 and get this error:

my code looks like this:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("test.db");

db.close();

my runtime args looks like this:
deno run -A --unstable test.ts

Did i miss something ?

needs better docs on how to query

Seems 0.5.x api has deprecated the old methods queryObject and queryArray. Can you put some examples in the docs on how to do this now?

empty string and NOT NULL constraint

Right now the following code:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database(":memory:");
db.exec(`
  CREATE TABLE foo (
    name TEXT NOT NULL
  );
`);

db.exec(`INSERT INTO foo (name) VALUES (?)`, "");

fails with:

error: Uncaught (in promise) Error: NOT NULL constraint failed: foo.name

I tried to figure out what's going on and seems like it's a problem with deno's ffi layer, where for an empty array it returns a null pointer:

~> deno --unstable
Deno 1.29.1
exit using ctrl+d, ctrl+c, or close()
REPL is running with all permissions allowed.
To specify permissions, run `deno repl` with allow flags.
> const b = new Uint8Array();
undefined
> Deno.UnsafePointer.of(b)
0

Which in turn when used with sqlite3_bind_text causes sqlite3 to bind null: https://github.com/sqlite/sqlite/blob/3547e4997f29afa0df561653019b795e6f8f691a/src/vdbemem.c#L1155

This is the code which causes a problem:

sqlite3/src/statement.ts

Lines 256 to 263 in 4a01a02

	case "string": {
	const str = (Deno as any).core.encode(param);
	this.#bindRefs.add(str);
	unwrap(
	sqlite3_bind_text(this.#handle, i + 1, str, str.byteLength, 0),
	);
	break;
	}

As a suggestion, perhaps something like this could work as a workaround:

// create a non-empty buffer globally somewhere
const emptyStrBuf = new Uint8Array(1);

// then when using sqlite3_bind_text in #bind check for empty string explicitly, 
// if it's empty use the buffer above, but specify zero length
if (param === "") {
   unwrap(
       sqlite3_bind_text(this.#handle, i + 1, emptyStrBuf, 0, 0),
   );
}

(using a non-empty array properly returns some valid pointer in deno)

> const b = new Uint8Array(1);
undefined
> Deno.UnsafePointer.of(b)
94900665479152

TS Error when running the vendored code

When running the vendored version of the code that uses this library, the user will get a TypeScript error described here:
denosaurs/plug#23

The issue was fixed in [email protected]. Could you consider upgrading the dependency, please?

`Failed to load SQLite3 Dynamic Library` – example code not working

I'm trying to run this code sample from readme:

import { Database } from "https://deno.land/x/[email protected]/mod.ts"

const db = new Database("test.db")

const [version] = db.prepare("select sqlite_version()").value<[string]>()!
console.log(version)

db.close()

And I'm getting this error:

error: Uncaught (in promise) Error: Failed to load SQLite3 Dynamic Library
  throw new Error("Failed to load SQLite3 Dynamic Library", { cause: e });
        ^
    at https://deno.land/x/[email protected]/src/ffi.ts:618:9
    at eventLoopTick (ext:core/01_core.js:183:11)
Caused by: TypeError: Deno.dlopen is not a function
    at dlopen (https://deno.land/x/[email protected]/mod.ts:155:15)
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async https://deno.land/x/[email protected]/src/ffi.ts:601:7

It might be somehow related to this:

• #89

I think of this relation because I'm also using aarch64 / Raspberry Pi. But it's another error so idk.

Version info:

deno --version
deno 1.36.0 (release, aarch64-unknown-linux-gnu)
v8 11.6.189.12
typescript 5.1.6

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

I already used dozens of other deno packages, they work perfectly fine.

Use tagged releases of SQLite

Currently the git submodule of sqlite not up to date nor using a tagged release. We should probably update to the latest version-3.40.1, and try to keep up with new tags as they are released.

Error running the example

example:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("test.db");

const [version] = db.prepare("select sqlite_version()").value<[string]>()!;
console.log(version);

deno run -A index.ts

Download https://github.com/denodrivers/sqlite3/releases/download/0.7.2/sqlite3.dll
error: Uncaught (in promise) Error: Failed to load SQLite3 Dynamic Library
  const error = new Error("Failed to load SQLite3 Dynamic Library");
                ^
    at https://deno.land/x/[email protected]/src/ffi.ts:566:17
Caused by: TypeError: request or response body error: error reading a body from connection: unexpected end of file
    at async readableStreamCollectIntoUint8Array (deno:ext/web/06_streams.js:822:21)
    at async consumeBody (deno:ext/fetch/22_body.js:232:11)
    at async protocolHttp (https://deno.land/x/[email protected]/file_fetcher.ts:25:18)
    at async fetchFile (https://deno.land/x/[email protected]/file_fetcher.ts:45:14)
    at async FileWrapper.fetch (https://deno.land/x/[email protected]/file.ts:95:18)
    at async FileWrapper.get (https://deno.land/x/[email protected]/file.ts:113:12)
    at async cache (https://deno.land/x/[email protected]/cache.ts:67:10)
    at async Wrapper.cache (https://deno.land/x/[email protected]/cache.ts:21:12)
    at async download (https://deno.land/x/[email protected]/plug.ts:107:16)
    at async prepare (https://deno.land/x/[email protected]/plug.ts:116:16)

deno v1.29.4

bind NaN got exception

(and it should be treat as null)

TypeError - Expected FFI argument to be a double, but got Null

at Object.opSync (deno:core/01_core.js:149:12)
at Object.DynamicLibrary.symbols. [as sqlite3_bind_double] (deno:ext/ffi/00_ffi.js:253:33)
at sqlite3_bind_double (https://deno.land/x/[email protected]/src/ffi.ts:495:30)
at PreparedStatement.bind (https://deno.land/x/[email protected]/src/database.ts:436:11)
at PreparedStatement.bindAll (https://deno.land/x/[email protected]/src/database.ts:515:12)

Getting this error for some reason

[uncaught application error]: TypeError - Invalid CString pointer, not valid UTF-8

request: { url: "http://briskreader.com/api/1/recent", method: "GET", hasBody: false }
response: { status: 404, type: undefined, hasBody: false, writable: true }

    at getColumn (https://deno.land/x/[email protected]/src/statement.ts:72:14)
    at eval (eval at #allNoArgs (https://deno.land/x/[email protected]/src/statement.ts:261:30), <anonymous>:6:20)
    at Statement.#allNoArgs [as all] (https://deno.land/x/[email protected]/src/statement.ts:423:19)
    at file:///home/ubuntu/www/briskreader.com/app/app.ts:248:5
    at dispatch (https://deno.land/x/[email protected]/middleware.ts:41:13)
    at https://deno.land/x/[email protected]/router.ts:1229:20
    at dispatch (https://deno.land/x/[email protected]/middleware.ts:41:13)
    at composedMiddleware (https://deno.land/x/[email protected]/middleware.ts:44:12)
    at dispatch (https://deno.land/x/[email protected]/router.ts:1235:28)
    at dispatch (https://deno.land/x/[email protected]/middleware.ts:41:13)

It's just doing a select:

	.get('/recent', async (context) => {
		const links = await context.state.db
			.prepare(
				'SELECT url, title, excerpt, created_at, count FROM urls ORDER BY created_at DESC LIMIT 50'
			)
			.all();

		context.response.body = links;
	})

use BigUint64Array for pointer to pointer

it's shorter and don't need to check endianess

  const pathPtr = cstr(path);
  const outDB = new BigUint64Array(1);

  const result = lib.symbols.sqlite3_open_v2(
    pathPtr,
    outDB,
    flags,
    new Deno.UnsafePointer(0n),
  ) as number;

  const handle = new Deno.UnsafePointer(outDB[0]);

sqlite3/src/ffi.ts

Lines 376 to 387 in 35c301f

	const pathPtr = cstr(path);
	const outDB = new Uint8Array(8);
	const outDV = new DataView(outDB.buffer);
	const result = lib.symbols.sqlite3_open_v2(
	pathPtr,
	outDB,
	flags,
	new Deno.UnsafePointer(0n),
	) as number;
	const handle = new Deno.UnsafePointer(outDV.getBigUint64(0, LITTLE_ENDIAN));

Open database readOnly not working

Hi there

I am trying to open the database in readOnly mode but it's not working.

My code:

const db = new Database("myfile.db", { readonly: true });

result:

error: Uncaught (in promise) SqliteError: 21: SQLite3 API misuse
    throw new SqliteError(code, "SQLite3 API misuse");
          ^
    at unwrap (https://deno.land/x/[email protected]/src/util.ts:36:11)
    at new Database (https://deno.land/x/[email protected]/src/database.ts:221:5)

V: 0.8.0

Thanks,

`ext/` folder containing import helpers for common extensions

An idea I had was to create an ext/ folder which contains helpers using plug for downloading and importing common extensions such as spatialite and the sqlean extension pack. This might be out of the scope of this repo or module and may be better off as it's own thing, especially considering these extensions may need to be built in the CI (sqlean already has binaries in it's repo, but spatialite would need building) but there are also ergonomic and ease-of-use benefits of keeping them here.

Support user-defined functions

SQLite's regex syntax errors out unless a user-defined regex function is defined.

User-defined functions are also useful for compensating for SQLite's poor datetime logic.

They can also be used with triggers to mimic Postgres' LISTEN/NOTIFY behavior.

I'd expect an implementation to resemble better-sqlite3's support for user-defined functions.

Segfault on debian aarch64 (raspberry pi)

Installed sqlite3 3.34.1 and ran:

export DENO_SQLITE_PATH="/usr/lib/aarch64-linux-gnu/libsqlite3.so"
deno run -A --unstable app.ts

Result is a segfault when trying to use any sqlite3 functionality

INTEGER type parsing incorrectly

Hi there, INTEGER type data doesn't seem to be getting parsed correctly. Here's how to recreate:
Table schema and data:

CREATE TABLE dates (dates INTEGER);
INSERT INTO dates (dates) VALUES (978307200000);
INSERT INTO dates (dates) VALUES (1009843200000);
INSERT INTO dates (dates) VALUES (1041379200000);
INSERT INTO dates (dates) VALUES (1072915200000);
INSERT INTO dates (dates) VALUES (1104537600000);

Via the CLI:

SELECT * FROM dates;

Returns, as expected:

978307200000
1009843200000
1041379200000
1072915200000
1104537600000

The following code:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("test.db");

const results = db.queryArray(`
  SELECT *
  FROM dates
`);

console.log(results);

db.close();

Returns, incorrectly:

[ [ -945343488 ], [ 525885440 ], [ 1997114368 ], [ -826624000 ], [ 731004928 ] ]

Deno version: 1.17.1 (release, x86_64-apple-darwin)
Module version: 0.2.3
macOS version: 12.1
SQLite version: 3.37.0

Invalid FFI pointer type, expected null, integer or BigInt

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database("./files/dist/northwind.db");

const a = db.query("SELECT * FROM \"Employees\"");
console.log(a);

db.close();

`ext/` folder containing import helpers for common extensions

An idea I had was to create an ext/ folder which contains helpers using plug for downloading and importing common extensions such as spatialite and the sqlean extension pack. This might be out of the scope of this repo or module and may be better off as it's own thing, especially considering these extensions may need to be built in the CI (sqlean already has binaries in it's repo, but spatialite would need building) but there are also ergonomic and ease-of-use benefits of keeping them here.

Add Backup API

https://www.sqlite.org/backup.html

Blob streaming

currently, entire blob is just read into memory and again copied into JS Uint8Array.
We could add "streaming" based blob reads using https://www.sqlite.org/c3ref/blob_open.html
to stream blob columns as ReadableStreams, or maybe could even investigate using Web Blob if possible.

Segfault after close() (regression from 0.6.1)

I've noticed the lib started to segfault after version 0.6.1 somewhere. My guess is that it's due to finalization code here:

sqlite3/src/database.ts

Lines 652 to 657 in a1c7e4d

	for (const [stmt, db] of STATEMENTS) {
	if (db === this.#handle) {
	sqlite3_finalize(stmt);
	STATEMENTS.delete(stmt);
	}
	}

Because it doesn't unregister global finalizers as it does here:

sqlite3/src/statement.ts

Lines 649 to 655 in a1c7e4d

	finalize(): void {
	if (!STATEMENTS.has(this.#handle)) return;
	this.#bindRefs.clear();
	statementFinalizer.unregister(this.#finalizerToken);
	STATEMENTS.delete(this.#handle);
	unwrap(sqlite3_finalize(this.#handle));
	}

A rough code sample that causes it:

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

const db = new Database(":memory:");
db.exec(`
  CREATE TABLE foo (
    id INTEGER NOT NULL
  );
`);

const s = db.prepare("INSERT INTO foo (id) VALUES (?)");
for (let i = 0; i < 1000000; i++) {
  s.run(i);
}

db.close();

const a: number[] = [];
for (let i = 0; i < 1000000; i++) {
  a.push(i);
}

console.log(a.reduce((s, v) => s + v, 0));

I actually don't know why it happens, but feels finalizer related. The random array code chunk at the end is important as it creates pressure on the GC.

Most likely "double free" happens somewhere.

Can somebody reproduce it?

uncaught error: invalid utf-8 sequence of 1 bytes from index 6

Error 'invalid utf-8 sequence of 1 bytes from index 6' contains boxed error of unknown type:
  Utf8Error { valid_up_to: 6, error_len: Some(1) }
error: Uncaught Error: invalid utf-8 sequence of 1 bytes from index 6
  return new Deno.UnsafePointerView(ptr).getCString();
                                         ^
    at Object.opSync (deno:core/01_core.js:149:12)
    at UnsafePointerView.getCString (deno:ext/ffi/00_ffi.js:106:19)
    at sqlite3_errmsg (https://deno.land/x/[email protected]/src/ffi.ts:330:42)
    at unwrap_error (https://deno.land/x/[email protected]/src/ffi.ts:339:17)
    at sqlite3_prepare_v3 (https://deno.land/x/[email protected]/src/ffi.ts:392:3)
    at Database.prepare (https://deno.land/x/[email protected]/src/database.ts:167:40)
    at Database.execute (https://deno.land/x/[email protected]/src/database.ts:143:25)
    at performJobs (file:///home/divy/Desktop/gh/deno_sqlite3/bench/x_sqlite3/sqlite3.ts:6:8)
    at performWorkflow (file:///home/divy/Desktop/gh/deno_sqlite3/bench/x_sqlite3/sqlite3.ts:17:5)
    at file:///home/divy/Desktop/gh/deno_sqlite3/bench/x_sqlite3/sqlite3.ts:31:19

Cannot reliably reproduce this but here's the code (same as in #1):

import { Database } from "https://deno.land/x/[email protected]/mod.ts";

db.execute("CREATE TABLE IF NOT EXISTS users (name TEXT NOT NULL)");

for (let i = 0; i < 10_000; i++) {
  db.execute("INSERT INTO users VALUES (?)", "littledivy");
}