GithubHelp home page GithubHelp logo

dallashoff / sqlocal Goto Github PK

View Code? Open in Web Editor NEW
143.0 6.0 7.0 508 KB

SQLocal makes it easy to run SQLite3 in the browser, backed by the origin private file system.

Home Page: https://sqlocal.dallashoffman.com

License: MIT License

TypeScript 100.00%
sqlite typescript-library vitest sqlite-wasm local-first

sqlocal's Introduction

SQLocal

SQLocal makes it easy to run SQLite3 in the browser, backed by the origin private file system. It wraps the WebAssembly build of SQLite3 and gives you a simple interface to interact with databases running on device.

Documentation - GitHub - NPM - Fund

Features

  • ๐Ÿ”Ž Locally executes any query that SQLite3 supports
  • ๐Ÿงต Runs the SQLite engine in a web worker so queries do not block the main thread
  • ๐Ÿ“‚ Persists data to the origin private file system, which is optimized for fast file I/O
  • ๐Ÿ”’ Each user can have their own private database instance
  • ๐Ÿš€ Simple API; just create a database and start running SQL queries
  • ๐Ÿ› ๏ธ Works with Kysely and Drizzle ORM for making type-safe queries

Examples

import { SQLocal } from 'sqlocal';

// Create a client with a name for the SQLite file to save in
// the origin private file system
const { sql } = new SQLocal('database.sqlite3');

// Use the "sql" tagged template to execute a SQL statement
// against the SQLite database
await sql`CREATE TABLE groceries (id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT)`;

// Execute a prepared statement just by inserting parameters
// in the SQL string
const items = ['bread', 'milk', 'rice'];
for (let item of items) {
  await sql`INSERT INTO groceries (name) VALUES (${item})`;
}

// SELECT queries and queries with the RETURNING clause will
// return the matched records as an array of objects
const data = await sql`SELECT * FROM groceries`;
console.log(data);

Log:

[
  { id: 1, name: 'bread' },
  { id: 2, name: 'milk' },
  { id: 3, name: 'rice' }
]

Or, you can use SQLocal as a driver for Kysely or Drizzle ORM to make fully-typed queries.

Kysely

import { SQLocalKysely } from 'sqlocal/kysely';
import { Kysely, Generated } from 'kysely';

// Initialize SQLocalKysely and pass the dialect to Kysely
const { dialect } = new SQLocalKysely('database.sqlite3');
const db = new Kysely<DB>({ dialect });

// Define your schema 
// (passed to the Kysely generic above)
type DB = {
  groceries: {
    id: Generated<number>;
    name: string;
  };
};

// Make type-safe queries
const data = await db
  .selectFrom('groceries')
  .select('name')
  .orderBy('name', 'asc')
  .execute();
console.log(data);

See the Kysely documentation for getting started.

Drizzle

import { SQLocalDrizzle } from 'sqlocal/drizzle';
import { drizzle } from 'drizzle-orm/sqlite-proxy';
import { sqliteTable, int, text } from 'drizzle-orm/sqlite-core';

// Initialize SQLocalDrizzle and pass the driver to Drizzle
const { driver } = new SQLocalDrizzle('database.sqlite3');
const db = drizzle(driver);

// Define your schema
const groceries = sqliteTable('groceries', {
  id: int('id').primaryKey({ autoIncrement: true }),
  name: text('name').notNull(),
});

// Make type-safe queries
const data = await db
  .select({ name: groceries.name })
  .from(groceries)
  .orderBy(groceries.name)
  .all();
console.log(data);

See the Drizzle ORM documentation for declaring your schema and making queries.

Install

Install the SQLocal package in your application using your package manager.

npm install sqlocal
# or...
yarn add sqlocal
# or...
pnpm install sqlocal

Cross-Origin Isolation

In order to persist data to the origin private file system, this package relies on APIs that require cross-origin isolation, so the page you use this package on must be served with the following HTTP headers. Otherwise, the browser will block access to the origin private file system.

Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin

If your development server uses Vite, you can do this by adding the following to your Vite configuration.

plugins: [
  {
    name: 'configure-response-headers',
    configureServer: (server) => {
      server.middlewares.use((_req, res, next) => {
        res.setHeader('Cross-Origin-Embedder-Policy', 'require-corp');
        res.setHeader('Cross-Origin-Opener-Policy', 'same-origin');
        next();
      });
    },
  },
],

Vite Configuration

Vite currently has an issue that prevents it from loading web worker files correctly with the default configuration. If you use Vite, please add the below to your Vite configuration to fix this.

optimizeDeps: {
  exclude: ['sqlocal'],
},

sqlocal's People

Contributors

dallashoff avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sqlocal's Issues

sqlocal easy connect/disconnect

I have a use case where I have to connect and disconnect on demand from databases.
Because of this I have created a wrapper to manage this, as a singleton.

I think it could be useful if the sqlocal client wouldn't actually connect to database on construction if fileName param was empty, and add the ability to connect on demand.

Any thoughts?

Database maximum size

Hello and thank you for this nice library.

I understand this is supposed to be run in the context of a web browser. With that in mind, how much data can the "DB" hold in practice. Meaning what is the maximum data size you effectively tested this library with while keeping responsive and practical.

Thank you again and best wishes!

no such function: REGEXP

Hello,

Many thanks for this great library!
It might be a lame question and unrelated to the SQLocal, but I can't figure out how to make the REGEXP operator work. My queries return with 'no such function: REGEXP.' error.
Thank you in advance for your help.

Error opening WAL db with overwriteDatabaseFile

Thanks for writing this library! it looks very clean.

When using overwriteDatabaseFile with a db that has WAL enabled I get the following error:
Error: SQLITE_CANTOPEN: sqlite3 result code 14: unable to open database file

Passing the byte array to the worker and using the sqlite3.oo1.OpfsDb.importDb('filename', byteArray);
function does allow it to be accessed.

Is this something that is open to being supported in the future?

Unable to parameterize table/column name in query

Have the following base query to drop table:

async drop () {
      return DB.sql`DROP TABLE IF EXISTS ${this.tableName}`;
};

which is creating following console error:

Error: sqlite3 result code 1: near "?": syntax error
    SQLite3Error http://127.0.0.1:5173/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs?v=83ec1d7f:5714
    toss3 http://127.0.0.1:5173/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs?v=83ec1d7f:10398
    checkSqlite3Rc http://127.0.0.1:5173/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs?v=83ec1d7f:10417
    checkRc http://127.0.0.1:5173/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs?v=83ec1d7f:10712
    exec http://127.0.0.1:5173/node_modules/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs?v=83ec1d7f:10842
    execQuery http://127.0.0.1:5173/node_modules/sqlocal/dist/worker.mjs?type=module&worker_file:47
    flushQueue http://127.0.0.1:5173/node_modules/sqlocal/dist/worker.mjs?type=module&worker_file:92
    init http://127.0.0.1:5173/node_modules/sqlocal/dist/worker.mjs?type=module&worker_file:124

checked the worker when db.exec is executed and confirmed the object is:

method: "all"
params: Array [ "Accounts" ]
queryKey: "snTm0m_khR34swnpiudaC"
sql: "DROP TABLE IF EXISTS ?"
type: "query"

Seems to be an issue with binding arguments to query. If I directly replace variable with text the query works fine.

Using nested literals to construct queries passed to SQLocal.sql

Not sure if appropriate question to post here but here goes.

Does anyone know if its possible to nest query template literals?
I cant seem to get it to work.
Example:

DB.sql<Transaction[]> `SELECT id, account_id, date, time, description, notes, amount, total, source, active, p_id, created_at, updated_at
        ${filter ? `WHERE description=${filter.description}` : ``}
        FROM Transactions`;

sqlite3 result code 10: disk I/O error

The browser throws the error below. Any ideas?

sqlite3-opfs-async-proxy.js?type=classic&worker_file:90 OPFS asyncer: xOpen DOMException: A requested file or directory could not be found at the time an operation was processed.
logImpl @ sqlite3-opfs-async-proxy.js?type=classic&worker_file:90
error @ sqlite3-opfs-async-proxy.js?type=classic&worker_file:94
xOpen @ sqlite3-opfs-async-proxy.js?type=classic&worker_file:645
await in xOpen (async)
f @ sqlite3-opfs-async-proxy.js?type=classic&worker_file:951
await in f (async)
globalThis.onmessage @ sqlite3-opfs-async-proxy.js?type=classic&worker_file:990
Show 5 more frames
Show less
sqlite3-bundler-friendly.mjs?v=6b26e52b:12388 OPFS syncer: xOpen() async error: NotFoundError: A requested file or directory could not be found at the time an operation was processed.
logImpl @ sqlite3-bundler-friendly.mjs?v=6b26e52b:12388
error @ sqlite3-bundler-friendly.mjs?v=6b26e52b:12392
opRun @ sqlite3-bundler-friendly.mjs?v=6b26e52b:12614
f @ sqlite3-bundler-friendly.mjs?v=6b26e52b:12958
$func507 @ sqlite3.wasm:0x24d5f
$func501 @ sqlite3.wasm:0x240d9
$sqlite3_open_v2 @ sqlite3.wasm:0x251ea
(anonymous) @ sqlite3-bundler-friendly.mjs?v=6b26e52b:7747
ctor @ sqlite3-bundler-friendly.mjs?v=6b26e52b:10589
OpfsDb @ sqlite3-bundler-friendly.mjs?v=6b26e52b:13174
value @ processor.js?v=6b26e52b:56
await in value (async)
value @ processor.js?v=6b26e52b:125
value @ processor.js?v=6b26e52b:91
self.onmessage @ worker.js?type=module&worker_file:4
Show 14 more frames
Show less
client.js?v=6b26e52b:56 Uncaught SQLite3Error: SQLITE_IOERR: sqlite3 result code 10: disk I/O error
at toss3 (sqlite3-bundler-friendly.mjs?v=6b26e52b:10492:17)
at checkSqlite3Rc (sqlite3-bundler-friendly.mjs?v=6b26e52b:10511:13)
at OpfsDb.ctor (sqlite3-bundler-friendly.mjs?v=6b26e52b:10591:13)
at new OpfsDb (sqlite3-bundler-friendly.mjs?v=6b26e52b:13174:45)
at SQLocalProcessor.value [as init] (processor.js?v=6b26e52b:56:35)
value @ client.js?v=6b26e52b:56
Show 1 more frame
Show less

Feature request: ability to control instantiation of sqlite db

Wow, amazing work here--really high quality code and docs, and just what I'm looking for (sqlite-wasm + Kysely).

That said, I wish the API supported a way for me to pass in the sqlite DB instead of sqlocal creating it (i.e., the ability to override SQLocalProcessor.init). Would you be open to making a change along those lines, or considering a PR to that effect?

Some reasons this might be a nice feature:

  • dev can control version of sqlite-wasm being used, including the possibility of packages other than the official @sqlite.org/sqlite-wasm distro (e.g., maybe one compiled with custom sqlite extensions, etc.)
  • dev has better control over storage mechanism (e.g., could use new sqlite.oo1.JsStorageDb(), or their own logic for how to handle OPFS not being available, etc.)
    • this also allows dev to control flags passed in to OpfsDb constructor (e.g., use trace flag in dev mode); although this could also be exposed as a param similar to databasePath

IndexedDB as a persistence option?

Hi,
SQLocale uses sqlite-wasm which to the best of my knowledge only uses OPFS for persisting data.
There could be other options. For instance absurd-sql uses IndexedDB for storing data.
Since OPFS is not widely and fully supported yet. Would you consider adding the option to use IndexedDB?
How much work would it be to let sqlocale wrap absurd-sql as an option?'
Thank you again!

Feature: Transaction

Firstly, thanks sincerely for your contributions.

Q. Are you interested in extending transactions to support the existing SQLite standard?

I am looking for a drop-in replacement for WebSQL/SQLite.
This library creates it's own unique interfaces [I mitigate by wrapping SQLocal.exec()].

Attempting to intercept executeSql() to manage transactions is complicated as inferred here: sqlite/sqlite-wasm#42 (comment)

The standard WebSQL/SQLite transaction interface is elegant; simply obtain a transaction object and use that.

db.transaction(
	(tx) => {
		tx.executeSql('INSERT INTO myTable', null, (tx, results) => {
			tx.executeSql('SELECT FROM myTable', null, (tx, results) => {});
		});
	},
	(error) => {}
);

Error during SvelteKit Server-side Rendering

I am using latest SvelteKit version (2.0.6), which uses bundler moduleResolution.
When i try to build or run vite dev on my project i am getting following error:

Cannot find module 'C:\sources\my-app\node_modules\sqlocal\dist\kysely\client' imported from C:\sources\my-app\node_modules\sqlocal\dist\kysely\index.js

I have created repository that reproduces issue when you run dev command. Unlike my actual project, build command seems to be working.
Any ideas how to solve it?

Node support

There seems to be a origin-private file-system ponyfill, it'd be cool if it could be used to make this module isomorphic and work everywhere with good performance, with the same APIs, and without ever touching native node modules.

How to handle adding new tables to the database when using drizzle?

I'm having problems connecting tables I define with my database. In the drizzle docs they mention migrations however it's not clear how to make this work with SQLocal.

My code follows the readme pretty closely. The database is created successfully but I can't insert anything into the table.

// db/index.ts
import "client-only";

import { drizzle } from "drizzle-orm/sqlite-proxy";
import { SQLocalDrizzle } from 'sqlocal/drizzle';

const { driver } = new SQLocalDrizzle('database.sqlite3');
const db = drizzle(driver);

// db/schema.ts
import { int, sqliteTable, text } from "drizzle-orm/sqlite-core";

export const groceries = sqliteTable("groceries", {
  id: int("id").primaryKey({ autoIncrement: true }),
  name: text("name").notNull(),
});

export type Groceries = typeof groceries.$inferSelect; // return type when queried

// component.ts
"use client"
import { database, sqlite } from "../db";
import { groceries } from "../db/schema";

...
database.insert(groceries).values([{ name: "milk" }, { name: "eggs" }]);

Binding in multiple statements within a query

Getting a binding error with following deletes within transaction
I had this working in the past with another sqlite3 package. Unsure what is failing.

IF I run each of the statements within the transaction individually it works fine.

return DB.sql`
                BEGIN TRANSACTION;
                    DELETE FROM BudgetTransactions WHERE budget_id IN (SELECT id FROM Budget WHERE id=${id} OR p_id=${id});
                    DELETE FROM BudgetCategories WHERE budget_id IN (SELECT id FROM Budget WHERE id=${id} OR p_id=${id});
                    DELETE FROM Budget WHERE id=${id} OR p_id=${id};
                COMMIT;
 `;

Error:

Uncaught (in promise) Error: Bind index 3 is out of range.

ERR_MODULE_NOT_FOUND when compiling for Node.js

There is some issue with importing drizzle (possibly related to #1?)
image

I have tried both "moduleResolution": "bundler" and "moduleResolution": "nodenext" as well as

    "paths": {
      "sqlocal/*": ["../node_modules/sqlocal/dist/*"]
    }

I also tried the minimal hello world example from your other repo, however to compile it I had to remove "noEmit": true and "allowImportingTsExtensions": true the problem persists :

โžœ  sqlocal-drizzle-hello-world git:(main) โœ— node src/main.js 

node:internal/modules/esm/resolve:264
    throw new ERR_MODULE_NOT_FOUND(
          ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/ondra/Documents/sqlocal-drizzle-hello-world/node_modules/.pnpm/[email protected]/node_modules/sqlocal/dist/drizzle/client' imported from /home/ondra/Documents/sqlocal-drizzle-hello-world/node_modules/.pnpm/[email protected]/node_modules/sqlocal/dist/drizzle/index.js
    at finalizeResolution (node:internal/modules/esm/resolve:264:11)
    at moduleResolve (node:internal/modules/esm/resolve:917:10)
    at defaultResolve (node:internal/modules/esm/resolve:1130:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:396:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:365:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:240:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:36) {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file:///home/ondra/Documents/sqlocal-drizzle-hello-world/node_modules/.pnpm/[email protected]/node_modules/sqlocal/dist/drizzle/client'
}

Node.js v20.11.1

How do you debug or visualize the database?

Hello,
I was wondering if there is a solution to "visualize" and debug the database tables (and the content)?
Maybe something like PhpMyAdmin or like the browser inspector for indexedDB.
Thank you for your help!

Concurrent reads and writes

Hello, first of all, thx for you wonderful lib, it's just amazing to use Drizzle ORM in my frontend ๐Ÿ˜…

I wonder the best way (most optimize way) to use your abstraction.
On each SQLocal instanciation, you create a new worker with a connection to the SQLite database. Thus, I wonder if we can have concurrent connection ? And concurrent reads ?

I use a Shared Worker in order to "store" the connection, and use that worker in other web workers when I need to retrieve some data. Should I create new connection for every worker which needs to access the database ?

Here is what I plan to do:

// Hack to work with Shared Workers in Typescript
const _self: {
  onconnect: (this: SharedWorkerGlobalScope, ev: MessageEvent<any>) => any;
} = self as any;
// Hack end

import { drizzle } from "drizzle-orm/sqlite-proxy";
import { SQLocalDrizzle } from "sqlocal/drizzle";

const db = drizzle(new SQLocalDrizzle("ashiso.sqlite3").driver);

//TODO: Apply migrations if needed

_self.onconnect = function (e) {
  var port = e.ports[0];

  port.addEventListener("message", function (e) {
    //TODO: Depends on the message type, execute appropriate query against the database
  });

  port.start();
};

What are your thoughts about that ??

Best regards,

Carere

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.