✌️ Single-file or PostgreSQL-server compatible transactional SQL database written in pure V.
Home Page: https://vsql.readthedocs.io
License: MIT License
vsql is a single-file or PostgreSQL-server compatible, transactional, SQL-compliant database written in pure V with zero dependencies.
After installing or updating, you can use vsql within your V applications, interact with database files using the CLI, or connect to a database using the built-in PostgreSQL-compatible server.
See the full documentation at vsql.readthedocs.io or use one of the quick links:
Thanks for this project - the prospect of another desktop db server is interesting to me (particularly if it support concurrency in the future - though I realise it's early days). I installed the server as a Windows exe (can't recall the version no - around the time it was publicised (around 1 Aug, which had the separate server & client exes) and that version of the server launched fine, but like Angel I could not connect with an .Net-based client so I left it for a while. I've revisited and downloaded the latest version, and upon launch the exe emits:
Unhandled Exception 0xC0000005
print_backtrace_skipping_top_frames_mingw is not implemented
This is JFYI.. If this is not fruitful, I can install the V compiler and try to poke around to see why this is happening (I've not touched V for a while).
It would be good for the server to emit the version number upon start-up.
In the latest version of V (8703e33) it cannot run tests with -prod - resulting in a segfault or inability to compile. I'm sure this will be fixed in the future. See Makefile.
Also, there was a related-ish bug that required PageObject to be public. See source for PageObject for more details.
Can you please provide an example on how to connect to the postgres server within the code itself as i plan for multiple people to use the program and the docs only said how to do it from command line.
The current paging implementation only allows objects to exist if they fit in a page. This is a serious limitation that will limit rows to be quite small.
CHAR(n) requires that the value be a fixed length and should be right padded with spaces. Actually I'm not sure about the padding, need to check that as well.
The page it points to doesn't exist, instead install instructions are located at https://vsql.readthedocs.io/en/latest/v-module.html
The linux binaries are currently disabled. See ci.yml for details.
Basically, the title says it all.
There should be a way to have the data on disk encrypted, for better security.
It would be useful to have a table with driver comparisons
At the moment the row IDs generated for storage have to rely on using the current timestamp and required a 1ms sleep to prevent collisions.
Running make bench fails after the SELECT stage of bench-on-disk
V full version: V 0.2.4 d4b3c65.d33f7d1
OS: Fedora linux
The TABLE ... statement is another form of SELECT * FROM ....
The header is always one page or 4kb (4096 bytes). At the moment only the first
byte is used for a rudimentary version. It will be expanded out in the future to
contain a magic number and other metadata for the file.
See "File Format"
Need to verify and fix any cases of different endianness between platforms.
As part of CI. In the mean time here is a way to build it:
VERSION=0.27.2
curl https://github.com/elliotchance/vsql/releases/download/v$VERSION/vsql-linux.zip --output vsql.zip
unzip vsql.zip
cat <<EOT >> Dockerfile
FROM alpine:3.14
RUN apk add gcompat
COPY vsql /usr/bin
ENTRYPOINT ["vsql"]
EOT
docker build -t vsql:$VERSION .
Interact with a file:
docker run -it vsql:$VERSION cli mydb.vsqlUnfortunaly, it looks like the vsql> prompt is misaligned with the input, so it does work but the formatting is all messed up:
elliotc@elliotc-C02FM0L1MD6R vsql % docker run -it vsql:$VERSION cli mydb.vsql
GC Warning: getcontext failed: using another register retrieval method...
vsql v0.27.2 2023-03-03
VALUES 1 + 2;
vsql> COL1: 3
1 row (11 ms)
VALUES 'Hello, World!';
vsql> COL1: Hello, World!
1 row (7 ms)
I'm not sure this is just a bug with the way it's packaged in docker.
You should also be able to start it as a PostgreSQL server with:
elliotc@elliotc-C02FM0L1MD6R vsql % docker run -it vsql:$VERSION server mydb.vsql
GC Warning: getcontext failed: using another register retrieval method...
vsql v0.27.2 2023-03-03
ready on 127.0.0.1:3210Although, I didn't try connecting to it.
Only integer types are supported at the moment.
BIGINT does not support this absolute minimum value because of the way the positive form (9223372036854775808) has to be encoded as an i64 before it is negated in the unary expression.
This problem should go away when NUMERIC types are added but make sure the bigint.sql test is updated before closing this issue.
Ordering by integers uses the left most expressions.
At the moment only one column is allowed.
Original posted by @dumblob. Moved here for easier discussion:
That depends on the transaction isolation probably. But generally it's impossible due to the SQL transactions nature which doesn't allow one to expect the whole content of the transaction up front (which would allow one to make this optimization).
But maybe vsql could introduce "by server steered transactions" instead of the typical "by client steered transactions". By server steered transactions would require the client to send the whole transaction (including client business logic inside of the transaction in the form of a set of data dependency graphs) to the server, then the server would examine it to determine whether certain optimizations could be done (by reading the business logic and trying to prove there is no dependency among the individual read values).
Then if there is any dependency, it wouldn't do any such optimization and instead call on the client to perform the business logic step. Then the client would report back, the server would perform the next lookup, send the result to client and so on until the end of the transaction. Of course if there wouldn't be any dependency, the server would prefetch all the requested values at once, commit the transaction and send all there prefetched values in one reply to the client which would then gradually use one by one of the received values as it would be grinding through the business logic.
Hm, thinking of this more - V actually allows one to do this due to AST capabilities. So the client could inspect the given transactional part of the AST and subsequently dependency graph (in compile time now when V has an interpreter, yay!) to be submitted to the server. Sounds too cool to be true.
But I'd guess this type of optimization has lots of potential as it'd finally allow one to write the code naturally instead of thinking how to structure it to split potentially bigger transaction into smaller chunks with different isolation preset and then tying all these fetched values manually and finally performing the rest of the desired business logic. As of now most of such transactions are either inefficient (because programmers are lazy) or they are unsafe (because programmers are even lazier and don't want to adjust their coding and instead they adjust e.g. the isolation level).
Thoughts?
This one is very important since a database may end up growing infinitely in size if we don't managed to make sure cleanup happens at least the rate of expired rows.
OS: linux, Ubuntu 22.04.1 LTS (WSL 2)
Processor: 4 cpus, 64bit, little endian, Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
CC version: cc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
getwd: /home/friday/projects/vsql-src/examples
vmodules: /home/friday/.vmodules
vroot: /home/friday/projects/v
vexe: /home/friday/projects/v/v
vexe mtime: 2022-12-11 14:27:37
is vroot writable: true
is vmodules writable: true
V full version: V 0.3.2 f5d0ba3.ad24c22
Git version: git version 2.34.1
Git vroot status: weekly.2022.49-22-gad24c222
.git/config present: true
thirdparty/tcc status: thirdparty-linux-amd64 12f392c3-dirty
v run simple.v
friday@DESKTOP-R96JRLH:~/projects/vsql-src/examples$ v simple.v
/home/friday/.vmodules/elliotchance/vsql/vsql/server.v:21:26: warning: reference field `elliotchance.vsql.vsql.Connection.options.query_cache` must be initialized (part of struct `elliotchance.vsql.vsql.ConnectionOptions`)
19 |
20 | pub fn new_server(options ServerOptions) Server {
21 | return Server{options, &Connection{
| ~~~~~~~~~~~
22 | query_cache: new_query_cache()
23 | }}
/home/friday/.vmodules/elliotchance/vsql/vsql/server.v:21:26: warning: reference field `elliotchance.vsql.vsql.Connection.options.mutex` must be initialized (part of struct `elliotchance.vsql.vsql.ConnectionOptions`)
19 |
20 | pub fn new_server(options ServerOptions) Server {
21 | return Server{options, &Connection{
| ~~~~~~~~~~~
22 | query_cache: new_query_cache()
23 | }}
/home/friday/.vmodules/elliotchance/vsql/vsql/cast.v:69:15: error: unknown method `cast_rules` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
67 | key := '$v.typ.typ AS $to.typ'
68 | if key in conn.cast_rules {
69 | return conn.cast_rules[key](conn, v, to)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
70 | }
71 |
/home/friday/.vmodules/elliotchance/vsql/vsql/eval.v:525:15: error: unknown method `binary_operators` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
523 | key := '$left.typ.typ $e.op $right.typ.typ'
524 | if key in conn.binary_operators {
525 | return conn.binary_operators[key](conn, left, right)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
526 | }
527 |
/home/friday/.vmodules/elliotchance/vsql/vsql/eval.v:536:15: error: unknown method `unary_operators` did you mean to access the field with the same name instead?.
12 possibilities: `find_function`, `open_read_connection`, `open_write_connection`, `prepare`, `query`, `register_func`, `register_function`, `register_virtual_table`, `release_read_connection`, `release_write_connection`, `schema_tables`, `schemas`.
534 | key := '$e.op $value.typ.typ'
535 | if key in conn.unary_operators {
536 | return conn.unary_operators[key](conn, value)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
537 | }
538 |
/home/friday/.vmodules/elliotchance/vsql/vsql/row.v:274:21: error: casting numbers to enums, should be done inside `unsafe{}` blocks
272 | match col.typ.typ {
273 | .is_boolean {
274 | v.bool_value = Boolean(buf.read_u8())
| ~~~~~~~~~~~~~~~~~~~~~~
275 | if v.bool_value == .is_unknown {
276 | v.is_null = true
SELECT * FROM t1
FETCH FIRST ROW ONLY;
-- X: 1Does not work, but is valid SQL.
There's currently no mitigation strategy for when transaction IDs wraparound. Even though the max transaction ID is high enough that most application won't ever encounter it, it needs to be fixed for v1.0.0.
v doctorOS: linux, Pop!_OS 22.04 LTS
Processor: 6 cpus, 64bit, little endian, AMD Ryzen 5 4500U with Radeon Graphics
CC version: cc (Ubuntu 11.2.0-19ubuntu1) 11.2.0
getwd: /home/astro/.vmodules/elliotchance/vsql
vmodules: /home/astro/.vmodules
vroot: /home/astro/Projects/V/v-main
vexe: /home/astro/Projects/V/v-main/v
vexe mtime: 2022-08-15 06:04:48
is vroot writable: true
is vmodules writable: true
V full version: V 0.3.0 5464de4.6fdcc5b
Git version: git version 2.34.1
Git vroot status: weekly.2022.32-34-g6fdcc5bc
.git/config present: true
thirdparty/tcc status: thirdparty-linux-amd64 827f7452
Here's the command I tried to execute:
make bin/vsql
I expected the project to compile with no errors.
Because of the recent deprecation, the following error are generated:
mkdir -p bin
v -prod cmd/vsql -o bin/vsql
vsql/pager.v:102:22: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
100 |
101 | mut buf := []u8{len: p.page_size}
102 | p.file.read(mut buf)?
| ^
103 |
104 | mut b := new_bytes(buf)
vsql/pg.v:80:24: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
78 | fn (mut c PGConn) read_byte() ?u8 {
79 | mut bytes := []u8{len: 1}
80 | c.conn.read(mut bytes)?
| ^
81 |
82 | return bytes[0]
vsql/pg.v:194:24: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
192 | fn (mut c PGConn) read_int32() ?int {
193 | mut bytes := []u8{len: 4}
194 | c.conn.read(mut bytes)?
| ^
195 |
196 | return int(binary.big_endian_u32(bytes))
vsql/pg.v:216:25: error: propagating a result like an option is deprecated, use `foo()!` instead of `foo()?`
214 | mut s := ''
215 | for {
216 | c.conn.read(mut bytes)?
| ^
217 | read++
218 | if bytes[0] == 0 {
make: *** [Makefile:18: bin/vsql] Error 1
This requires a Pull Request that updates the current code to match the updated syntax.
First I'd like to say I like the decision to strictly conform to a (modern) SQL standard. I think this might be one of the selling points - "it was tested against vsql" would have a huge meaning in the bright future I envision 😉.
Here I'd like to discuss probably the biggest PITA any DB has to clearly decide and define. It's durability guarantees, i.e. a "fault model". This also determines the maximum achievable performance of the DB due to physical limitations.
So I always envisioned the following:
to have a DB API requiring one to specify on per-request basis whether it shall be treated as best-effort (i.e. no guarantee but presumably faster and with lower delays) or as guaranteed operation. See e.g. my rant redis/redis#6200 (comment) .
Note that best effort might to lead incorrect data compared to the surrounding global context but it must not lead to internally incorrect data within a transaction - e.g. if in one transaction I fetch two different rows from two different tables it still must guarantee that the data (i.e. both rows) returned are exactly the data from the point in time when the transaction began (i.e. mutually correct).
Guarantee durability in the very maximum technically possible way POSIX allows us to and the knowledge of HW-originated errors allows us to. Namely the same fault model as TigerBeetle has. SQLite3 allows almost for what TigerBeetle does (e.g. commit both the parent directory structure as well as the DB file itself and then fsync) but there are still some rough corners.
The first step would be to focus on calling fsync and waiting for it to finish (see below) after each transaction which has the potential to modify data.
Note also that Linux fsync is synchronous (and will stay so in the upcoming years) but on many other systems it's asynchronous with system-specific calls to wait for its return.
What do you think about implementing both points (1) and (2)?
This breaks because * doesn't take into account extra columns
I'm not sure if this is possible during tests? If not a separate build stage to build all the binaries in stricter -prod mode.
vsql>
vsql> create table products (title character varying(100), price float);
msg: Option('CREATE TABLE 1')
1 row (0 ms)
vsql> INSERT INTO products (title, price) VALUES ('Instant Pot', 144.89);
cli execution error: cannot create lock file sample2.vsql.lockHi was trying to create a table with 'name' as a column but got a crash, looking into the lexer i realized that name is marked as a keyword. Is this by design?
vsql> create table person (name character varying(100));
# cli execution error: vsql.SQLState42601: syntax error: near "NAME"<comment> ::= <simple comment>
| <bracketed comment>
<simple comment> ::=
<simple comment introducer> [ <comment character>... ] <newline>
<simple comment introducer> ::=
<minus sign> <minus sign>
<bracketed comment> ::=
<bracketed comment introducer>
<bracketed comment contents>
<bracketed comment terminator>
<bracketed comment introducer> ::=
/*
<bracketed comment terminator> ::=
*/
<bracketed comment contents> ::=
[ { <comment character> | <separator> }... ] !! See the Syntax Rules.
A response can contain a message, like "CREATE TABLE 1". However, right now this comes back as a fake table response instead which confuses the protocol and user. A response needs to have a dedicate field for this.
CREATE TABLE t1 (x INT, b INT, PRIMARY KEY(x));
EXPLAIN SELECT * FROM t1 WHERE x = 10 AND b = 20;Parser errors should be way more friendly, they should at least:
VALUES is a helpful shorthand that doesn't require a temporary table to be created if you just want to SELECT an expression.
There is an in-memory pager available as part of the v0.12.0 release but there is no way to enable it. Perhaps special ":memory:" file name like SQLite3 does?
The n is currently ignored. The standard isn't really clear about what we are supposed to do here:
- FLOAT specifies the data type approximate numeric, with binary precision equal to or greater than the value of the specified . The maximum value of is implementation-defined. shall not be greater than this value.
Add vsql version command. Also show the version when running the server.
Sequences are the SQL standard way of doing auto increment, serials, etc.
An optional escape character can be specified:
<character like predicate part 2> ::=
[ NOT ] LIKE <character pattern> [ ESCAPE <escape character> ]
<similar predicate part 2> ::=
[ NOT ] SIMILAR TO <similar pattern> [ ESCAPE <escape character> ]
Support for INSERT with multiple VALUES.
Running the example produces 0, instead of the inserted values:
import elliotchance.vsql.vsql
fn main() {
mut db := vsql.open('/tmp/test.vsql') ?
// All SQL commands use query():
db.query('CREATE TABLE foo (a FLOAT)') ?
db.query('INSERT INTO foo (a) VALUES (1.23)') ?
db.query('INSERT INTO foo (a) VALUES (4.56)') ?
// Iterate through a result:
result := db.query('SELECT * FROM foo') ?
for row in result {
println(row.get_f64('a'))
}
// See SQLSTATE (Errors) below for more examples.
}The values are inside the /tmp/test.vsql as validated by running the vsql-cli.v later.
Build binary releases of vsql-cli and vsql-server so that V isn't required to use the database if someone just wanted to play with it.
psql is a common client that should work but I'm seeing a timeout during the connection. Perhaps a message that's not implemented correctly?
The following test does not work because we currently use i64 for all ints. Add these when this is no longer a limitation:
real.sql
CREATE TABLE foo (x REAL);
INSERT INTO foo (x) VALUES (123456789123456789123456789);
SELECT CAST(x AS BIGINT) FROM foo;
-- msg: CREATE TABLE 1
-- msg: INSERT 1
-- error 22003: numeric value out of rangedouble-precision.sql
CREATE TABLE foo (x DOUBLE PRECISION);
INSERT INTO foo (x) VALUES (123456789123456789123456789);
SELECT CAST(x AS BIGINT) FROM foo;
-- msg: CREATE TABLE 1
-- msg: INSERT 1
-- error 22003: numeric value out of rangeThere was a garbage collection bug that force me to change server mode into a single connection. I believe with the changes to boehm this problem should be gone now.
This one is pretty important...
Hello there!
vsql looks like a very cool project and I wanted to play with it (particularly the server part) I tried to connect with a .NET driver and a jdbc driver but I was not lucky at all no exceptions were reported by the server
I understand these are the early days, so perhaps I'm a bit too early?
The underlying walker does support range queries, but the planner does not understand how to set the lower and upper bounds, like:
CREATE TABLE t1 (x INT, PRIMARY KEY(x));
EXPLAIN SELECT * FROM t1 WHERE x > 10;Currently all builtin databases support the orm, and it would be nice if this one does too.
Needed for support:
orm.Connection
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.