kkharji / sqlite.lua

SQLite LuaJIT binding with a very simple api.

sqlite.lua 💫

SQLite/LuaJIT binding and a highly opinionated wrapper for storing, retrieving, caching, and persisting SQLite databases. sqlite.lua present new possibilities for plugin development and while it's primarily created for neovim, it support all luajit environments.

• Changelog
• Docs
• Examples
• Powered By sqlite.lua

✨ Features:

• Connect, reconnect, close sql db connections sqlite:open/sql:close
• Evaluate any sqlite statement and return result if any sqlite:eval
• Helper function over sqlite:eval to do all sort of operation.
• High level API with sqlite.tbl for better experience.
• lua tables deserialization/serialization (in helper functions and high level api)
• 90% test coverage.
• Up-to-date docs and changelog

🚧 Installation

Packer.nvim (Neovim)

use { "kkharji/sqlite.lua" }

luarocks (LuaJIT)

luarocks install sqlite luv

Ensure you have sqlite3 installed locally. (if you are on mac it might be installed already)

Windows

Download precompiled and set let g:sqlite_clib_path = path/to/sqlite3.dll (note: /)

Linux

sudo pacman -S sqlite # Arch
sudo apt-get install sqlite3 libsqlite3-dev # Ubuntu
sudo dnf install sqlite sqlite-devel # Fedora

Nix (home-manager)

programs.neovim.plugins = [
    {
      plugin = pkgs.vimPlugins.sqlite-lua;
      config = "let g:sqlite_clib_path = '${pkgs.sqlite.out}/lib/libsqlite3.so'";
    }
];

Notes:

• Ensure you install pkgs.sqlite
• If you are using home-manager on OSX, you must replace libsqlite3.so with libsqlite3.dylib

🔥 Powered by sqlite.lua

• https://github.com/kkharji/impatient.nvim
• https://github.com/nvim-telescope/telescope-smart-history.nvim
• https://github.com/nvim-telescope/telescope-frecency.nvim
• https://github.com/kkharji/lispdocs.nvim
• https://github.com/nvim-telescope/telescope-cheat.nvim

sqlite.lua's Issues

(extend) db.init doesn't set db.is_initialized to true

changelog: find a way to render links with collapse or remove

88f14bf: auto changelog ([#80](https://github.com//issues/80))

Here goes nothing 🤞. Please CI don't fail me.

should be

88f14bf: auto changelog (#80)

configuring sqlite pargma results in db lock error

The current way by which pargma or sqlite database configuration get set, is causing the following error msg

sql.nvim: couldn't parse sql statement, ERRMSG: database is locked.

Perhaps using __wrap_stmts would fix this ...

The current way:

    if self.sqlite_opts then
      for k,v in pairs(self.sqlite_opts) do
        if not u.valid_pargma_key(k) then
          return error("sql.nvim: " .. k .. " is not a valid pragma")
        end
        clib.exec(self.conn, string.format("pragma %s = %s", k, v), nil, nil, nil)
      end
    end

fix change logs when the commit is refering to an issue

auto changelog (#80)

Wheres entries should allow table to support more flexibility

where should allow something like this

db:get{
  tbl = {
    where = {
      id = { 1, 2, 3, 4, 5, 6 } -- this table is new
    }
  }
}

sql:Insert should fall back to sql:update when a row exists

This may be a bit convenient and also follow the way in which lua tables get mutated by setting new value to them.

changelog: run once a day

should all db/sql method ensure valid connection?

Should all db methods ensure like sql.with_open that the db connection is open? and should we introuduce additional parameter that if provided the connection should be closed after this method is executed?

(extend) support adding custom table name

support setting a different table name than the provided key. e.g.

require('sql') {
  init = true,
  fs = {
      _name = "filesystem_table"
  }
}
fs:insert {...} -- will insert into table named "filesystem_table" and not "fs"

better support for boolean and lua tables

Currently sql.nvim uses the following for interoping lua boolean to 0/1.

-- lua/sql.lua
local booleansql = function(value)
 if type(value) == "boolean" then
   value = (value == true) and 1 or 0
 end
 return value
end

However there are no support for interop 0/1 back to lua boolean. This aspect
should be handled in stmt module with the above code removed from sql:eval.

sql:insert and table:insert should not require the primary key, instead should return it.

Hello just testing the project, looks great just have some comments on the insert behavior.
stmt:step returns SQLITE_ROW after an insert from where the id of the inserted row is retrieved, there's also last_insert_row_id. Therefore the primary key of a table shouldn't be required when inserting.

Each entry in most SQLite tables (except for WITHOUT ROWID tables) has a unique 64-bit signed integer key called the "rowid". The rowid is always available as an undeclared column named ROWID, OID, or ROWID as long as those names are not also used by explicitly declared columns. If the table has a column of type INTEGER PRIMARY KEY then that column is another alias for the rowid.

bulk insert breaks if one of the dict has missing key

Inserting multiple rows or dicts results in error `wrong number of argumentsq if one of the dicts has a missing key.

table cache tests fails 3/10 of the time, race condition?

I'm not sure whether I should just comment these tests out or find out why this is happening.

any ideas

fix sql.lua docs not being generated :( #56

#56

(extend) Merge table function and set warnings

When creating un initialized db, table functions defined before initialization get removed. Additionally, table fields under db would be nil when trying to extend them or add function to them.

Thus, an empty table should be in place and at initialization merge the result of table:extend with what has been already defined.

Note: This maybe tricky with __new_index and __index setup of sql.table. I may need to modify the function to add fields to self instead of sql.table

foregin keys support & tests

schema = { ..., foreign_keys = { client = "projects.id", } } == "KEY(client) REFERENCES projects(id)""

In addition to auto enable.

extend should use db and tbl instead of super

super is a bit wired key for an object that can bd extended to be anything.

(table.extend) custom methods get overwritten to pass self as first param

create and hold db object in variables with opening connection.

Sometimes we'd like to create a db object and hold it a variable without opening a db connection.

One option is to pass true as second argument to sql.open, the other is to have addtional method sql.new to do that.

Which one makes more sense?

auto alter table schema

It's really annoying to have to call table drop or keep track of schema.ensure. In nvim-telescope/telescope-frecency.nvim#35, I was trying to debug and understand why count field keep being nil, even though the schema has default=0. It turns out the calling files.schema() returned an old schema, which means that the new schema has not effect on pre-existing table.

I'd like to compare the sql table schema and user provided schema and compare
between them, and if they are different. I'd like table module to auto alter.

The full table schema information as it exists in C side can be found with db:schema("files", true), it output something like that

{
  def = {},
  info = {
    count = { cid = 0, primary = false, required = false, type = "INTEGER" },
    id = { cid = 1, primary = true, required = false, type = "INTEGER" },
    path = { cid = 2, primary = false, required = false, type = "TEXT" }
  },
  req = {},
  types = { count = "INTEGER", id = "INTEGER", path = "TEXT" }
}

In telescope-frecency case, I wanted it to change to the following:

{
  def = { count = "0" },
  info = {
    count = { cid = 0, default = "0", primary = false, required = false, type = "integer" },
    id = { cid = 1, primary = true, required = true, type = "integer" },
    path = { cid = 2, primary = false, required = false, type = "string" }
  },
  req = {},
  types = { count = "integer", id = "integer", path = "string" }
}

delete/remove is failing silently

#65 same issue

Add auto changelog

Use github actions scheduled jobs to auto generate CHANGELOG.md once every two days.

update is failing silently with foreign key

during tests, update with foreign key enabled should have resulted in a failure but it fail silently and new data wasn't reflect in the database

libsqlite3.so fails to load on nix/guix (workaround)

This probably isn't pertinent to most people, but I'm posting this in case others run into this issue. libz will fail to load before libsqlite3.so is loaded, so you get an error message about libsqlite3 when it's really the fault of libz.

I'm going to try to figure out how to deal with this on the nix packaging side. The following is for neovim installed on a linux system (not nixos), but it should work with the store path's substituted for the appropriate libraries.

diff --git a/lua/sql/defs.lua b/lua/sql/defs.lua
index edc260a..31de4ef 100644
--- a/lua/sql/defs.lua
+++ b/lua/sql/defs.lua
@@ -3,13 +3,8 @@ local bit = require'bit'
 
 local M = {}
 
-local clib_path = vim.g.sql_clib_path or (function()
-  if vim.loop.os_uname().sysname == 'Darwin' then
-    return '/usr/local/opt/sqlite3/lib/libsqlite3.dylib'
-  end
-  return 'libsqlite3'
-end)()
-local clib = ffi.load(clib_path)
+ffi.load('/usr/lib64/libz.so')
+local clib = ffi.load('/usr/lib64/libsqlite3.so')
 
 -- Constants
 M.flags = {

table insert should just return id or nil

tables should have methods of their own

Data and sql tables might be eaiser to deal if they can have a set of method.

Policies:

• Connection is opend or not shouldn't be an issue. Additionally, if one of the functions gets called with closed connection then the connection will be opend and closed after the operations is done.
• Results should be cached with the the query and deleted on insert/update/delete operations.

Specifications

sql:table

Initialize the sql table object with:

• exists,
• has_content,
• schema,
• last_action,
• last_insert_id, -- hmmm I wonder if thats even important
• cache (when for some reason we had to call for all the rows)

local t = db:table("todos")

t:schema

Create or change schema of the table. If the schema doesn't have ensure key, then the table will be droped and created with the new schema. Unless, the table already has data, then the it should error out.

t:schema{
  id = {"integer", "not", "null"},
  name = "text",
  age = "integer"
}

t:add

same functionalities as sql.insert

t:add{ 
  id = 1, 
  name = "a" 
}

t:add{ 
  {id = 1, name = "b" }, 
  {id = 3, name = "c", age = 19} 
}

t:remove

same functionalities as sql.delete. if no arguments is provided then it should remove all rows.

t:remove() -- remove everything
t:remove{ id = 1, name = "a" }
t:remove{ id = {1,2,3,4}, name = "a" }
t:remove{ contains = { name = "tam" }}

t:replace

same functionalities as sql.insert, but delete the content of table and replaced with the content.

t:replace{
  -- rows
}

t:get

same functionalities sql:select.

t:get{
  keys = { "name" },
  where = { 
    id = 1 
    name = function() return "do stuff and return a string" end
  },
  contains = { name = "ab" },
  -- join
}

t:update

same functionalities as sql.update

t:update{
  where = { .. }
  values = { .. },
}

t:update{
  {where = { .. } values = { .. }},
  {where = { .. } values = { .. }},
  {where = { .. } values = { .. }},
}

t:drop

drop the table if it exists and delete content if any. Else do nothing.

t:drop()

t:empty

returns not t.has_content

t:empty()

t:exists

returns t.exists

t:exists()

t:each

Iterates over a table rows of with a function that get execute on each row.

t:each({
  where = { .. }
  contains = { .. },
}, function(row) 
  -- execute stuff
end)

t:map

produce a new table by a function. uses t:each.

local modified = t:map({
  where = { .. }
  contains = { .. },
}, function(row) 
  -- execute stuff
  -- return modified a new row or nothing
end)

t:count`:

Returns the size or count of rows, used to set has_content

t:sort`:

Returns sorted rows based on comparison function, if not function
is provided and key is then it sorts by that key value

cc @Conni2461

sql.table.insert should error out or skip nil rows

Make exending and creating dsl from sql/table object easy

(table.where) returns nil on existing row

For some reason tbl.where return nil.

Maybe it is something todo with the stupid dot notations or inherently a bug with
how where works.

Enforce emmylua classes, type and return types for autocompletion

support open_v2

No description provided.

merge where and contain keys in parser.lua

if any of string in where key has UNIX wildcards then it should be placed in glob clause rather than where clause

fix table module cache

There few failing tests for caching that I need to fix or remove entirely

table.each accept function as first arg

support table.each/map to accept function as first argument and query as second argument and vice-versa

optional cache for table module

sql.delete and sql.table.remove should only accept where cluase

deprecation warning should be put in place for now and still support it

sql:close should finalize statements if any

(extend) accept pre-made tables

With #92 merged. Users can define independent tables then inject db in them. Thus, db:extend need to accept pre-made table and call set_db on them.

sql:create should accept a table of schemas

No description provided.

:insert{table, table} doesn't work as expected.

Reproduce:

local table1 = { ... }
local table2 = { ... }
local db = sql.open(path)
db:insert{ table1, table2 }
assert.are.same("table", type(db:eval("select * from table1")))

After doing the insertation indvidiually, It errored out that one of the table
had undefined field in the sqlite table schema. However, after fixing that and
trying to re-run insert{}, same issue.

Expected

• It errors out as it would with indvidiual insertation.
• It insert multiple row in multiple tables successfully.

Suggestions

• Check if all table fields are vaild fields based on the table schema.

sql:insert doesn't respect default values

When a schema defines default values, the current implementation doesn't respect that and somehow endup inserting null in those fields instead.

Example:

create table if not exists test(
  id integer primary key,
  title text not null,
  created datetime default current_timestamp,
)

sqlite function in insert break other values

Apparently the following parsed statement is not good enough to pass:

insert into timestamps (fid, date) values(:fid, julianday('now'))"

it does insert the date, but sadly ignore fid.

🤔 not sure how this would be solved 😭

function DB:insert(tbl_name, rows)
  --- ...
  clib.wrap_stmts(self.conn, function()
    for _, v in ipairs(items) do
      local s = stmt:parse(self.conn, P.insert(tbl_name, { values = v }))
      --> P.insert(....) => "insert into timestamps (fid, date) values(:fid, julianday('now'))"
      --> stmt:parse(..) => "insert into timestamps (fid, date) values(NULL, julianday('now'))"
      s:bind(v)
      s:step()
      s:bind_clear()
      table.insert(ret_vals, s:finalize())
    end
  end)
end

On the parser side, instead of binding to :date it uses the value in place.

local pvalues = function(defs, kv)
  --- ....
  local keys = {}
  for k, v in u.opairs(defs) do
    if type(v) == "string" and v:match "%a+%(.+%)" then
      tinsert(keys, v)
    else
      tinsert(keys, ":" .. k)
    end
  end

  return ("values(%s)"):format(tconcat(keys, ", "))
end

(table) query/remove/update using index access

table.insert/update/remove passing functions

As of now, pass sqlite builtin functions as string insert them as strings

try uv before vim.loop

There are few places where we use vim.loop, we should have pacall to uv before loop. Also if we can't replace vim.fn.fnamemodify in utill then we should just remove since it will be rarely where plugin developers try to load db from current path.

fix parser.lua failing test #56

#56

Fail	||  parse [create] ensure that the table is created with a key being true	
            parser_spec.lua:126: should be identical
            Expected objects to be the same.
            Passed in:
            (string) 'create table if not exists people(age int, id integer not null primary key, name text)'
            Expected:
            (string) 'create table if not exists people(age int, id integer primary key, name text)'
            
            stack traceback:
               parser_spec.lua:117>

README references missing docs/sql.txt

title

Create a set of examples

• Web Bookmark system example
• Local Git Repos tracker example
• Todo manager example

overwriting schema may cause duplications when calling table:schema

This might be better commented out

        self.tbl_schema = self.db:schema(self.name)
        return self.tbl_schema

sql.table.insert doesn't return id

Not sure but while writing tests in #56 the function didn't return id