elliotchance / vsql

✌️ Single-file or PostgreSQL-server compatible transactional SQL database written in pure V.

✌️ vsql

vsql is a single-file or PostgreSQL-server compatible, transactional, SQL-compliant database written in pure V with zero dependencies.

After installing or updating, you can use vsql within your V applications, interact with database files using the CLI, or connect to a database using the built-in PostgreSQL-compatible server.

See the full documentation at vsql.readthedocs.io or use one of the quick links:

• FAQ
• CLI
• Supported PostgreSQL clients
• Features
• SQL Reference
• Contributing
• Functions
• SQLSTATE
• SQL Compliance

vsql's Issues

SELECT *, ABS(10 - x) FROM foo

This breaks because * doesn't take into account extra columns

sql: PRIMARY KEY with multiple columns

At the moment only one column is allowed.

Objects may extend beyond a single page

The current paging implementation only allows objects to exist if they fit in a page. This is a serious limitation that will limit rows to be quite small.

Support for SEQUENCES

Sequences are the SQL standard way of doing auto increment, serials, etc.

ORDER BY 1, 2, 3

Ordering by integers uses the left most expressions.

INSERT bulk

Support for INSERT with multiple VALUES.

Transaction questions

Original posted by @dumblob. Moved here for easier discussion:

That depends on the transaction isolation probably. But generally it's impossible due to the SQL transactions nature which doesn't allow one to expect the whole content of the transaction up front (which would allow one to make this optimization).

But maybe vsql could introduce "by server steered transactions" instead of the typical "by client steered transactions". By server steered transactions would require the client to send the whole transaction (including client business logic inside of the transaction in the form of a set of data dependency graphs) to the server, then the server would examine it to determine whether certain optimizations could be done (by reading the business logic and trying to prove there is no dependency among the individual read values).

Then if there is any dependency, it wouldn't do any such optimization and instead call on the client to perform the business logic step. Then the client would report back, the server would perform the next lookup, send the result to client and so on until the end of the transaction. Of course if there wouldn't be any dependency, the server would prefetch all the requested values at once, commit the transaction and send all there prefetched values in one reply to the client which would then gradually use one by one of the received values as it would be grinding through the business logic.

Hm, thinking of this more - V actually allows one to do this due to AST capabilities. So the client could inspect the given transactional part of the AST and subsequently dependency graph (in compile time now when V has an interpreter, yay!) to be submitted to the server. Sounds too cool to be true.

But I'd guess this type of optimization has lots of potential as it'd finally allow one to write the code naturally instead of thinking how to structure it to split potentially bigger transaction into smaller chunks with different isolation preset and then tying all these fetched values manually and finally performing the rest of the desired business logic. As of now most of such transactions are either inefficient (because programmers are lazy) or they are unsafe (because programmers are even lazier and don't want to adjust their coding and instead they adjust e.g. the isolation level).

Thoughts?

Allow multiple connections in server mode

There was a garbage collection bug that force me to change server mode into a single connection. I believe with the changes to boehm this problem should be gone now.

Cannot compile with `-prod`

In the latest version of V (8703e33) it cannot run tests with -prod - resulting in a segfault or inability to compile. I'm sure this will be fixed in the future. See Makefile.

Also, there was a related-ish bug that required PageObject to be public. See source for PageObject for more details.

sql: PRIMARY KEY to support range queries

The underlying walker does support range queries, but the planner does not understand how to set the lower and upper bounds, like:

CREATE TABLE t1 (x INT, PRIMARY KEY(x));
EXPLAIN SELECT * FROM t1 WHERE x > 10;

Ensure CHAR(n) are space padded

CHAR(n) requires that the value be a fixed length and should be right padded with spaces. Actually I'm not sure about the padding, need to check that as well.

Build binary releases

Build binary releases of vsql-cli and vsql-server so that V isn't required to use the database if someone just wanted to play with it.

Improve parser errors

Parser errors should be way more friendly, they should at least:

1. Describe the exact position (line/column number)
2. Show more surrounding tokens or the whole line so its much easier to understand where the error is.
3. Provide helpful tips for common misconceptions. For example, inline PRIMARY KEY should provide a link to the doc with a example rather than just say "error at PRIMARY"

examples not work with latest v version

v doctor

OS: linux, Ubuntu 22.04.1 LTS (WSL 2)
Processor: 4 cpus, 64bit, little endian, Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
CC version: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

getwd: /home/friday/projects/vsql-src/examples
vmodules: /home/friday/.vmodules
vroot: /home/friday/projects/v
vexe: /home/friday/projects/v/v
vexe mtime: 2022-12-11 14:27:37
is vroot writable: true
is vmodules writable: true
V full version: V 0.3.2 f5d0ba3.ad24c22

Git version: git version 2.34.1
Git vroot status: weekly.2022.49-22-gad24c222
.git/config present: true
thirdparty/tcc status: thirdparty-linux-amd64 12f392c3-dirty

v run simple.v

friday@DESKTOP-R96JRLH:~/projects/vsql-src/examples$ v  simple.v 
/home/friday/.vmodules/elliotchance/vsql/vsql/server.v:21:26: warning: reference field `elliotchance.vsql.vsql.Connection.options.query_cache` must be initialized (part of struct `elliotchance.vsql.vsql.ConnectionOptions`)
   19 | 
   20 | pub fn new_server(options ServerOptions) Server {
   21 |     return Server{options, &Connection{
      |                             ~~~~~~~~~~~
   22 |         query_cache: new_query_cache()
   23 |     }}
/home/friday/.vmodules/elliotchance/vsql/vsql/server.v:21:26: warning: reference field `elliotchance.vsql.vsql.Connection.options.mutex` must be initialized (part of struct `elliotchance.vsql.vsql.ConnectionOptions`)
   19 | 
   20 | pub fn new_server(options ServerOptions) Server {
   21 |     return Server{options, &Connection{
      |                             ~~~~~~~~~~~
   22 |         query_cache: new_query_cache()
   23 |     }}
/home/friday/.vmodules/elliotchance/vsql/vsql/cast.v:69:15: error: unknown method `cast_rules` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
   67 |     key := '$v.typ.typ AS $to.typ'
   68 |     if key in conn.cast_rules {
   69 |         return conn.cast_rules[key](conn, v, to)
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   70 |     }
   71 |
/home/friday/.vmodules/elliotchance/vsql/vsql/eval.v:525:15: error: unknown method `binary_operators` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
  523 |     key := '$left.typ.typ $e.op $right.typ.typ'
  524 |     if key in conn.binary_operators {
  525 |         return conn.binary_operators[key](conn, left, right)
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  526 |     }
  527 |
/home/friday/.vmodules/elliotchance/vsql/vsql/eval.v:536:15: error: unknown method `unary_operators` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
  534 |     key := '$e.op $value.typ.typ'
  535 |     if key in conn.unary_operators {
  536 |         return conn.unary_operators[key](conn, value)
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  537 |     }
  538 |
/home/friday/.vmodules/elliotchance/vsql/vsql/row.v:274:21: error: casting numbers to enums, should be done inside `unsafe{}` blocks
  272 |             match col.typ.typ {
  273 |                 .is_boolean {
  274 |                     v.bool_value = Boolean(buf.read_u8())
      |                                    ~~~~~~~~~~~~~~~~~~~~~~
  275 |                     if v.bool_value == .is_unknown {
  276 |                         v.is_null = true

SELECT FROM VALUES

VALUES is a helpful shorthand that doesn't require a temporary table to be created if you just want to SELECT an expression.

'name' is reserved: cant use 'name' as a column name

Hi was trying to create a table with 'name' as a column but got a crash, looking into the lexer i realized that name is marked as a keyword. Is this by design?

vsql> create table person (name character varying(100));
# cli execution error: vsql.SQLState42601: syntax error: near "NAME"

Allow in-memory database

There is an in-memory pager available as part of the v0.12.0 release but there is no way to enable it. Perhaps special ":memory:" file name like SQLite3 does?

Improve file format header

The header is always one page or 4kb (4096 bytes). At the moment only the first
byte is used for a rudimentary version. It will be expanded out in the future to
contain a magic number and other metadata for the file.

See "File Format"

Add benchmarks with other drivers (sqlite, pg)

It would be useful to have a table with driver comparisons

Dropping a table does not delete rows

This one is pretty important...

planner: Understand PK in logical operation

CREATE TABLE t1 (x INT, b INT, PRIMARY KEY(x));
EXPLAIN SELECT * FROM t1 WHERE x = 10 AND b = 20;

Fix transaction ID wraparound

There's currently no mitigation strategy for when transaction IDs wraparound. Even though the max transaction ID is high enough that most application won't ever encounter it, it needs to be fixed for v1.0.0.

-9223372036854775808 overflows

BIGINT does not support this absolute minimum value because of the way the positive form (9223372036854775808) has to be encoded as an i64 before it is negated in the unary expression.

This problem should go away when NUMERIC types are added but make sure the bigint.sql test is updated before closing this issue.

CI should use "-prod"

I'm not sure if this is possible during tests? If not a separate build stage to build all the binaries in stricter -prod mode.

LIKE and SIMILAR TO do not support ESCAPE

An optional escape character can be specified:

<character like predicate part 2> ::=
    [ NOT ] LIKE <character pattern> [ ESCAPE <escape character> ]

<similar predicate part 2> ::=
    [ NOT ] SIMILAR TO <similar pattern> [ ESCAPE <escape character> ]

sql: PRIMARY KEY to support more types

Only integer types are supported at the moment.

Support TABLE statement

The TABLE ... statement is another form of SELECT * FROM ....

psql is not working

psql is a common client that should work but I'm seeing a timeout during the connection. Perhaps a message that's not implemented correctly?

Recover expired rows in MVCC

This one is very important since a database may end up growing infinitely in size if we don't managed to make sure cleanup happens at least the rate of expired rows.

File format is not cross platform

Need to verify and fix any cases of different endianness between platforms.

make bin/vsql fails due to distinction between result and option

Output of v doctor

OS: linux, Pop!_OS 22.04 LTS
Processor: 6 cpus, 64bit, little endian, AMD Ryzen 5 4500U with Radeon Graphics
CC version: cc (Ubuntu 11.2.0-19ubuntu1) 11.2.0

getwd: /home/astro/.vmodules/elliotchance/vsql
vmodules: /home/astro/.vmodules
vroot: /home/astro/Projects/V/v-main
vexe: /home/astro/Projects/V/v-main/v
vexe mtime: 2022-08-15 06:04:48
is vroot writable: true
is vmodules writable: true
V full version: V 0.3.0 5464de4.6fdcc5b

Git version: git version 2.34.1
Git vroot status: weekly.2022.32-34-g6fdcc5bc
.git/config present: true
thirdparty/tcc status: thirdparty-linux-amd64 827f7452

The Command

Here's the command I tried to execute:

make bin/vsql

The Expected Result

I expected the project to compile with no errors.

The Actual Result

Because of the recent deprecation, the following error are generated:

mkdir -p bin
v  -prod cmd/vsql -o bin/vsql
vsql/pager.v:102:22: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
  100 | 
  101 |     mut buf := []u8{len: p.page_size}
  102 |     p.file.read(mut buf)?
      |                         ^
  103 | 
  104 |     mut b := new_bytes(buf)
vsql/pg.v:80:24: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
   78 | fn (mut c PGConn) read_byte() ?u8 {
   79 |     mut bytes := []u8{len: 1}
   80 |     c.conn.read(mut bytes)?
      |                           ^
   81 | 
   82 |     return bytes[0]
vsql/pg.v:194:24: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
  192 | fn (mut c PGConn) read_int32() ?int {
  193 |     mut bytes := []u8{len: 4}
  194 |     c.conn.read(mut bytes)?
      |                           ^
  195 | 
  196 |     return int(binary.big_endian_u32(bytes))
vsql/pg.v:216:25: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
  214 |     mut s := ''
  215 |     for {
  216 |         c.conn.read(mut bytes)?
      |                               ^
  217 |         read++
  218 |         if bytes[0] == 0 {
make: *** [Makefile:18: bin/vsql] Error 1

The Solution

This requires a Pull Request that updates the current code to match the updated syntax.

CAST(REAL AS BIGINT) boundary checking

The following test does not work because we currently use i64 for all ints. Add these when this is no longer a limitation:

real.sql

CREATE TABLE foo (x REAL);
INSERT INTO foo (x) VALUES (123456789123456789123456789);
SELECT CAST(x AS BIGINT) FROM foo;
-- msg: CREATE TABLE 1
-- msg: INSERT 1
-- error 22003: numeric value out of range

double-precision.sql

CREATE TABLE foo (x DOUBLE PRECISION);
INSERT INTO foo (x) VALUES (123456789123456789123456789);
SELECT CAST(x AS BIGINT) FROM foo;
-- msg: CREATE TABLE 1
-- msg: INSERT 1
-- error 22003: numeric value out of range

Provide docker image

As part of CI. In the mean time here is a way to build it:

VERSION=0.27.2

curl https://github.com/elliotchance/vsql/releases/download/v$VERSION/vsql-linux.zip --output vsql.zip
unzip vsql.zip

cat <<EOT >> Dockerfile
FROM alpine:3.14
RUN apk add gcompat
COPY vsql /usr/bin
ENTRYPOINT ["vsql"]
EOT

docker build -t vsql:$VERSION .

Interact with a file:

docker run -it vsql:$VERSION cli mydb.vsql

Unfortunaly, it looks like the vsql> prompt is misaligned with the input, so it does work but the formatting is all messed up:

elliotc@elliotc-C02FM0L1MD6R vsql % docker run -it vsql:$VERSION cli mydb.vsql
GC Warning: getcontext failed: using another register retrieval method...
vsql v0.27.2 2023-03-03
VALUES 1 + 2;
vsql> COL1: 3 
1 row (11 ms)

VALUES 'Hello, World!';
vsql> COL1: Hello, World! 
1 row (7 ms)

I'm not sure this is just a bug with the way it's packaged in docker.

You should also be able to start it as a PostgreSQL server with:

elliotc@elliotc-C02FM0L1MD6R vsql % docker run -it vsql:$VERSION server mydb.vsql
GC Warning: getcontext failed: using another register retrieval method...
vsql v0.27.2 2023-03-03
ready on 127.0.0.1:3210

Although, I didn't try connecting to it.

Support SQL comments

<comment> ::= <simple comment>
  | <bracketed comment>

<simple comment> ::=
  <simple comment introducer> [ <comment character>... ] <newline>

<simple comment introducer> ::=
  <minus sign> <minus sign>

<bracketed comment> ::=
  <bracketed comment introducer>
      <bracketed comment contents>
      <bracketed comment terminator>

<bracketed comment introducer> ::=
  /*

<bracketed comment terminator> ::=
  */

<bracketed comment contents> ::=
[ { <comment character> | <separator> }... ] !! See the Syntax Rules.

server: Cannot connect using .NET driver

Hello there!
vsql looks like a very cool project and I wanted to play with it (particularly the server part) I tried to connect with a .NET driver and a jdbc driver but I was not lucky at all no exceptions were reported by the server

I understand these are the early days, so perhaps I'm a bit too early?

"FETCH FIRST ROW ONLY"

SELECT * FROM t1
FETCH FIRST ROW ONLY;
-- X: 1

Does not work, but is valid SQL.

Row ID generation needs to be improved

At the moment the row IDs generated for storage have to rely on using the current timestamp and required a 1ms sleep to prevent collisions.

Fix linux build

The linux binaries are currently disabled. See ci.yml for details.

Fix "msg" in response

A response can contain a message, like "CREATE TABLE 1". However, right now this comes back as a fake table response instead which confuses the protocol and user. A response needs to have a dedicate field for this.

Readme points to the wrong place for install instructions

The page it points to doesn't exist, instead install instructions are located at https://vsql.readthedocs.io/en/latest/v-module.html

make bench fails with panic `more than two versions is not allowed`

Running make bench fails after the SELECT stage of bench-on-disk

V full version: V 0.2.4 d4b3c65.d33f7d1
OS: Fedora linux

Durability (& reliability) versus performance

First I'd like to say I like the decision to strictly conform to a (modern) SQL standard. I think this might be one of the selling points - "it was tested against vsql" would have a huge meaning in the bright future I envision 😉.

Here I'd like to discuss probably the biggest PITA any DB has to clearly decide and define. It's durability guarantees, i.e. a "fault model". This also determines the maximum achievable performance of the DB due to physical limitations.

So I always envisioned the following:

1. to have a DB API requiring one to specify on per-request basis whether it shall be treated as best-effort (i.e. no guarantee but presumably faster and with lower delays) or as guaranteed operation. See e.g. my rant redis/redis#6200 (comment) .

   Note that best effort might to lead incorrect data compared to the surrounding global context but it must not lead to internally incorrect data within a transaction - e.g. if in one transaction I fetch two different rows from two different tables it still must guarantee that the data (i.e. both rows) returned are exactly the data from the point in time when the transaction began (i.e. mutually correct).

2. Guarantee durability in the very maximum technically possible way POSIX allows us to and the knowledge of HW-originated errors allows us to. Namely the same fault model as TigerBeetle has. SQLite3 allows almost for what TigerBeetle does (e.g. commit both the parent directory structure as well as the DB file itself and then fsync) but there are still some rough corners.

The first step would be to focus on calling fsync and waiting for it to finish (see below) after each transaction which has the potential to modify data.

Note also that Linux fsync is synchronous (and will stay so in the upcoming years) but on many other systems it's asynchronous with system-specific calls to wait for its return.

What do you think about implementing both points (1) and (2)?

Support for builtin orm

Currently all builtin databases support the orm, and it would be nice if this one does too.

Needed for support:

• implement orm.Connection
• get it hardcoded into the V codebase

vsql should support transparent data encryption (TDE)

Basically, the title says it all.

There should be a way to have the data on disk encrypted, for better security.

v0.11.0 server crash upon start

Thanks for this project - the prospect of another desktop db server is interesting to me (particularly if it support concurrency in the future - though I realise it's early days). I installed the server as a Windows exe (can't recall the version no - around the time it was publicised (around 1 Aug, which had the separate server & client exes) and that version of the server launched fine, but like Angel I could not connect with an .Net-based client so I left it for a while. I've revisited and downloaded the latest version, and upon launch the exe emits:

Unhandled Exception 0xC0000005
print_backtrace_skipping_top_frames_mingw is not implemented

This is JFYI.. If this is not fruitful, I can install the V compiler and try to poke around to see why this is happening (I've not touched V for a while).

It would be good for the server to emit the version number upon start-up.

Failure to create lock file

vsql>
vsql> create table products (title character varying(100), price float);
msg: Option('CREATE TABLE 1')
1 row (0 ms)

vsql> INSERT INTO products (title, price) VALUES ('Instant Pot', 144.89);
cli execution error: cannot create lock file sample2.vsql.lock

PG server!

Can you please provide an example on how to connect to the postgres server within the code itself as i plan for multiple people to use the program and the docs only said how to do it from command line.

example shows 0s, instead of the inserted data

Running the example produces 0, instead of the inserted values:

import elliotchance.vsql.vsql

fn main() {
	mut db := vsql.open('/tmp/test.vsql') ?

	// All SQL commands use query():
	db.query('CREATE TABLE foo (a FLOAT)') ?
	db.query('INSERT INTO foo (a) VALUES (1.23)') ?
	db.query('INSERT INTO foo (a) VALUES (4.56)') ?

	// Iterate through a result:
	result := db.query('SELECT * FROM foo') ?
	for row in result {
		println(row.get_f64('a'))
	}

	// See SQLSTATE (Errors) below for more examples.
}

The values are inside the /tmp/test.vsql as validated by running the vsql-cli.v later.

Add version command

Add vsql version command. Also show the version when running the server.

`FLOAT(n)` does not have any affect

The n is currently ignored. The standard isn't really clear about what we are supposed to do here:

29. FLOAT specifies the data type approximate numeric, with binary precision equal to or greater than the value of the specified . The maximum value of is implementation-defined. shall not be greater than this value.