gluesql / gluesql

GlueSQL is quite sticky. It attaches to anywhere.

GlueSQL

Multi-Model Database Engine as a Library

GlueSQL is a Rust library for SQL databases that includes a parser (sqlparser-rs), an execution layer, and a variety of storage options, both persistent and non-persistent, all in one package. It is a versatile tool for developers, supporting both SQL and its own query builder (AST Builder). GlueSQL can handle structured and unstructured data, making it suitable for a wide range of use cases. It is portable and can be used with various storage types, including log files and read-write capable storage. GlueSQL is designed to be extensible and supports custom planners, making it a powerful tool for developers who need SQL support for their databases or services. GlueSQL is also flexible, as it can be used in Rust and JavaScript environments, and its language support is constantly expanding to include more programming languages.

For more information on how to use GlueSQL, please refer to the official documentation website. The documentation provides detailed information on how to install and use GlueSQL, as well as examples and tutorials on how to create custom storage systems and perform SQL operations.

"We offer a service where the GlueSQL team can implement and maintain your custom storage, especially beneficial for NoSQL databases with their own query planner and execution layer. We welcome any services wishing to support SQL and GlueSQL query interfaces. For more details, please refer to here."

If you're interested in learning more about GlueSQL, we recommend the following blog articles for a deeper dive into its capabilities and benefits:

1. Breaking the Boundary between SQL and NoSQL Database
2. Revolutionizing Databases by Unifying Query Interfaces
3. Test-Driven Documentation - Automating User Manual Creation

Supporting SQL and AST Builder

GlueSQL supports both SQL and its own query builder (AST Builder). Unlike other ORMs, GlueSQL's AST Builder allows developers to build queries directly with GlueSQL's AST, enabling the use of all of GlueSQL's features. This is why we named it AST Builder instead of Query Builder.

Rust Example

• example: pkg/rust/examples/hello_world.rs

let storage = MemoryStorage::default();
let mut glue = Glue::new(storage);

let payloads = glue.execute("SELECT * FROM Foo;").await.unwrap();

SQL Example

SELECT id, name FROM Foo WHERE name = 'Lemon' AND price > 100

AST Builder Example

table("Foo")
    .select()
    // Filter by name using a SQL string
    .filter("name = 'Lemon'")
    // Filter by price using AST Builder methods
    .filter(col("price").gt(100))
    .project("id, name")
    .execute(&mut glue)
    .await;

Supporting Structured and Unstructured Data with Schema Flexibility

GlueSQL supports both structured and unstructured (schemaless) data. While SQL databases typically assume that schemas are defined and used, GlueSQL does not make this assumption. It supports completely unstructured data, similar to a NoSQL document database, as well as semi-structured types such as MAP and LIST. This makes GlueSQL suitable for a wide range of use cases, including those that require handling of unstructured data. Additionally, it is possible to join tables with schemas and schemaless tables together and execute queries.

Schemaless SQL Example

CREATE TABLE Names (id INTEGER, name TEXT);
INSERT INTO Names VALUES (1, 'glue'), (2, 'sql');

CREATE TABLE Logs;
INSERT INTO Logs VALUES
    ('{ "id": 1, "value": 30 }'),
    ('{ "id": 2, "rate": 3.0, "list": [1, 2, 3] }'),
    ('{ "id": 3, "rate": 5.0, "value": 100 }');

SELECT * FROM Names JOIN Logs ON Names.id = Logs.id;
/*
| id | list    | name | rate | value |
|----|---------|------|------|-------|
| 1  |         | glue |      | 30    |
| 2  |[1, 2, 3]| sql  | 3    |       |
*/

Supported Reference Storages

GlueSQL provides a variety of reference storages out of the box, including simple in-memory storage, key-value databases, log file-based storage like JSON & JSONL, and even Web Storage and IndexedDB supported by web browsers. These reference storages are readily available for use and can be easily adapted to a variety of storage systems. Additionally, GlueSQL is constantly expanding its list of supported storages, making it a versatile tool for developers.

Memory Storage

Memory Storage is a foundational storage option designed for in-memory, non-persistent data. It is a simple yet robust storage option that can be used in production environments.

Shared Memory Storage

Shared Memory Storage is a storage option designed to provide more comfortable usage of Memory Storage in concurrent environments. It wraps the Memory Storage with a read-write lock and an atomic reference count, allowing you to clone the storage instance and use it effortlessly across multiple threads. All storage instances will refer to the same data, making it a convenient option for concurrent environments.

Sled Storage

Sled Storage is a persistent data storage option for GlueSQL that is built on the Sled key-value embedded database in Rust. It is the only storage option currently supported by GlueSQL that implements all Store traits, from non-clustered indexes to transactions. Sled Storage is an excellent choice for handling and storing data in a Rust environment. To use Sled Storage, you can create a SledStorage instance using a path.

JSON Storage

With GlueSQL, you can use JSONL or JSON files as a database that supports SQL and AST Builder, making it a powerful option for developers who need to work with JSON data. JSON Storage is a storage system that uses two types of files: a schema file (optional) and a data file. The schema file is written in Standard SQL and stores the structure of the table, while the data file contains the actual data and supports two file formats: *.json and *.jsonl. JSON Storage supports all DML features, but is particularly specialized for SELECT and INSERT.

Mongo Storage

With Mongo storage, you can use mongodb as a storage for SQL queries. You can use all the features supported by GlueSQL, such as aggregations and joins, which were previously difficult to handle on an unstructured database. In particular, you can use GlueSQL's powerful schema system on mongodb, which is as strong as an RDBMS. To run tests, refer to here

Web Storage

WebStorage, specifically localStorage and sessionStorage, can be used as a data storage system for GlueSQL. While WebStorage is a simple key-value database that uses string keys, GlueSQL makes it more powerful by adding support for SQL queries. This allows you to use SQL to interact with WebStorage, making it a convenient option for developers who are familiar with SQL. WebStorage can be used in JavaScript (Web) environments and Rust WebAssembly environments.

IndexedDB Storage

IndexedDB Storage is a powerful storage system that allows you to interact with IndexedDB using SQL. While using IndexedDB directly can be challenging, GlueSQL makes it easy to use by handling version management internally and storing data in JSON format. With GlueSQL, you can use SQL to interact with IndexedDB, making it a convenient option for developers who are familiar with SQL. You can use IndexedDB Storage in both JavaScript (Web) and Rust WebAssembly environments.

Composite Storage

Composite Storage is a powerful feature of GlueSQL that allows you to bundle together multiple existing storages, enabling you to perform JOIN operations across two distinct storages. This feature is utilized in various environments, including GlueSQL's JavaScript (Web) interface. Specifically, GlueSQL bundles together memory, localStorage, sessionStorage, and IndexedDB using Composite Storage in its JavaScript (Web) interface. This allows you to create tables using four different storages and perform operations like JOIN using SQL, all at once. Composite Storage is a versatile feature that can be used in many different scenarios, making it a valuable tool for developers who need to work with multiple storage systems, including those that require data migration between different storage systems.

Adapting GlueSQL to Your Environment: Creating Custom Storage

GlueSQL is designed to be adaptable to a wide variety of environments, including file systems, key-value databases, complex NoSQL databases, and remote APIs. To create a custom storage for GlueSQL, you only need to implement the Store and StoreMut traits provided by GlueSQL. These traits allow you to support SELECT queries and modify data, such as INSERT, UPDATE, and DELETE.

If you want to support additional features, such as schema changes, transactions, or custom functions, you can implement the corresponding traits. However, these traits are optional, and you can choose to implement only the ones that are relevant to your storage system.

To make it even easier to develop custom storages, GlueSQL provides a Test Suite that allows you to test your storage implementation against a set of standard SQL queries. This ensures that your storage system is compatible with GlueSQL and can handle common SQL operations.

Overall, creating a custom storage for GlueSQL is a straightforward process that allows you to adapt SQL and the AST Builder to your environment with ease.

GlueSQL Custom Storage: Let Us Handle It for You

Although anyone can develop a custom storage for GlueSQL with ease, our GlueSQL team can also implement and maintain it for you. This is especially recommended for NoSQL databases with their own query planner and execution layer, as adapting GlueSQL to them requires a deep understanding of GlueSQL's planner and storage layer details. We welcome not only database companies but also any services that want to support SQL and GlueSQL query interfaces. As GlueSQL is rapidly adding and improving features, we can help you develop and manage your custom storage effectively if you entrust it to us. If you're interested, please contact us at [email protected].

Contribution

GlueSQL is a database project that is simpler than you might think. You only need to know three common Rust project commands: cargo fmt, cargo clippy, and cargo test. Don't hesitate to make pull requests and change the code as you see fit. We have set up GitHub Actions to validate your changes, so you don't have to worry about making mistakes. The line coverage of GlueSQL's core code is almost 99%, which is the result of not only careful test writing, but also of making the test suite easy to understand and use for anyone, even those who are not familiar with Rust. If you're not sure where to start, we recommend exploring the test suite first. Take a look at the existing features and try to understand how they work. Even if you're not familiar with Rust, you should be able to navigate the test suite without any problems. If there's a feature you'd like to see but isn't there yet, implementing it yourself and contributing it to GlueSQL is a great way to get involved. You can also check out the issues on the GlueSQL GitHub repository for more ideas on how to contribute.

License

This project is licensed under the Apache License, Version 2.0 - see the LICENSE file for details.

gluesql's Issues

SELECT foo AS bar ... support,

mentioned in #80

SELECT id AS SOMEALIASCOLUMNNAME, num FROM Test

Current blend.rs is not handling AS.
Simple fix, blend does not need to work on AS at all, just ignore AS and simply evaluate column expression.

BETWEEN support

SELECT * FROM TableA WHERE amount BETWEEN 1 AND 100

MVCC Transactions Support

I can only imagine how large a feature this is but I wanted to put it out there. MVCC Transaction support would empower developer's to prevent reading and writing inconsistent data to gluesql.

I'm guessing however that this is currently blocked on Sled's lack of transactional scan support inside TransactionTrees(?)

RIGHT [OUTER] JOIN support

For now, GlueSQL supports 2 join types, INNER JOIN and LEFT OUTER JOIN.
RIGHT OUTER JOIN Implementation can be a bit more tough, but all modifications required to change is in a single file, 198 lines of codes.

• src/executor/join.rs

It can be a sort of quite fun programming quiz.

Fail to find table alias to blend in SELECT + JOIN query

SELECT user.* FROM item LEFT JOIN user ON user.id = item.user_id;
SELECT * FROM item LEFT JOIN user ON user.id = item.user_id;

Above 2 queries work but,

SELECT Item.* FROM item LEFT JOIN user ON user.id = item.user_id;

This one fails.
[ERR] { "Blend": { "TableNotFound": "item" } }
Blend failed to find table alias.

DROP TABLE query support

DROP Table Something;

Replace "Context" data structure from LinkedList to Vector

GlueSQL fetches data and convert each row into struct named Context.

FilterContext contains the row from join and nested select.
BlendContext does similar role but it does not need to include nested selected row on where clause.

Currently linked list is used but it's better to replace those into vector.
Linked list assumes that total number of items are not predictable, but in this case, it is not.
It is certainly pre-calculated from AST made from sql query.

So, it means it's possible to specify the length of context at first, like using this.. Vec::with_capacity(10).

But let's start with simply replacing data structure, from linked list to vector.
At this point, it would be good to start with using im-rc

This is not urgent issue, but it should be done in... some day.

Enhance programmer methods on Select result

Have made some programming around select.
It's a little bit complicated and column names is missing I think.

In order to make glusql competitive to JS, sqlite, PHP and more
I think something like the following is required
Some kind of Displayselected instead od Selected
The outermost SELECT could get this result.
But of course not for an INSERT .. SELECT ...

1. Column names
2. Simplified enums on cell values

A: Usage is: iter over colnames in order to display them in order
B: Usage on whole row is: 'simple' item for all column values
C: Usage on columns in a row

let cell = arow[colname.iter().position(|&x| x == "THECOLNAME")

Below is some pseudo Rust for data definitions

All is immutable

enum DisplayCell
{
Str {value:&str},
Int {value:i64},
Float {value:f64},
Empty, // null
Bool {value:&str} // 'true', 'false' or 1,0 for language independancy
}
struct DisplayRow
{
arow:Vec
}

struct Displayselected
{
headers:Vec<&str>,
rows:Vec
}

think about adding github releases

It would be very nice for people like me who would like to be notified about version updates, without necessarily having to be notified about issues and PRs 🥰

Getting SELECT projection aliases

Related to #80 - due to long threads, it's hard to recognize what exact task is, so let's use this issue.

input:

SELECT a, b AS foo, SUM(id) AS bar FROM TableA;

output:

vec!["a", "foo", "bar"]

This issue is adding a new function get_projection_aliases to src/parse.rs which takes sqlparser::ast::Query reference and return column aliases, Vec<String>

fn get_projection_aliases(query: &sqlparser::ast::Query) -> Vec<String>

Valid Columntypes not checked at CREATE TABLE

CREATE TABLE foolish (first INTEGER, second ANYTHING_ACCEPTED);

Perhaps
Checks in execute.rs
fn prepare
and match Statement::CreateTable { name, columns, .. }

Add more types to Value

pub enum Value {
    I64(i64),
    String(String),
}

Now GlueSQL only has 2 value types.
Let's add two more types,

• Bool(bool)
• Real(f64)

Then It looks enough for the first release.

Replace sql parser from nom-sql to sqlparser-rs

Use this,
https://github.com/andygrove/sqlparser-rs

SELECT 1 FROM Test; not working

SELECT 1 + 1 FROM Test;

Above works, but...

SELECT 1 FROM Test;

This doesn't work with this error [Blend Error] "FieldDefinitionNotSupported"

Cannot insert null to nullable text type field

In src/data/value.rs,

current from_data_type function is not handling String type, unit test required

ORDER BY support

Hej,

Sorry for all issues. Not complaining. Love the SQL and in Rust!

But browsing into my swedish testdabase on recipes and it's categories
the output is not ordered on column 2.

(We have the same sort order in Swedish as in any Latin-1 countries)

"select * from category where categoryid > 50 order by 2;"
53 "Fläskkött" ""
54 "Vegetarisk" ""
55 "Pastej" ""
56 "Gratäng" ""
57 "Finskt" ""
58 "Kassler" ""

Plus(+) & Minus(-) signs support in WHERE

Following WHERE clauses are not supported now.

SELECT * FROM TableA WHERE +1 = 1;
SELECT * FROM TableA WHERE -1 = -1;

Proper place to add test cases would be in src/tests/filter.rs.

DEFAULT column attribute support for CREATE TABLE

CREATE TABLE Foo (
  id INTEGER DEFAULT 1
);

For now, create sql query is accepted but not working.
Modification in row.rs is required.

'SingleQuotedString' is not accepted in inserts

Problem with insert of text.

Insert into tableA values ("WORKS BUT NOT STANDARD" , 'STANDARD BUT IS NOT ACCEPTED');

Most exports from other databases uses SingleQuotedString. More Standard SQL.

Seems like expressions with SingleQuotedString works

COUNT, MIN, MAX, SUM supports

• COUNT
• MIN
• MAX
• SUM

Let's get it.

Remove panic! uses in evaluated.rs

src/executor/evaluate/evaluated.rs

There are a few panic!() uses in evaluated.rs.
I used panic!() because currently there's no way to reach that branches.
But it's better not to use panic, replace those to managed unreachable error types.

Single quoted string is not accepted

INSERT INTO TableA VALUES ("asdf");

INSERT INTO TableA VALUES ('asdf');

"asdf" works but 'asdf' is not accepted which should be not a problem at all.

CLI interface

Is a command line interface planned? I liked playing around with the web interface but there doesn't seem to be something equivalent for the terminal?

Column names in Select results

Have made some programming around select in Rust
It's a little bit complicated and column names is missing I think.

In order to make gluesql competitive to (sqlite and mysql and more ) using (JS, C/C++, PHP, Java and more)
I think something like the following is required for Rust at least
(and C using gluesql. wow!)

1. Array of column names on payload

A: Usage is: iter over colnames in order to display them in order
B: Usage on whole row is: 'simple' item for all column values
C: Usage on columns in a row
I think it's important to get a value for a named column in a 'active row'
(From my own experience)

let value = get_value_from_column_named(colnam, some_context);

Testdata from file and/or stdin

Love this project.
Have a lot of sql database dumps I want to have for verification
A separate bin project can't refer to locally libraries.

But a rough approach is to introduce tests for loading sql from a file instead of inline strings

Give me a hint where to start in my downloaded glusql-main

GROUP BY and HAVING support

Comparing to Join, it's just a simple task.
Let's implement it before the first release!

Specify rule for UpdateError::ConflictOnSchema like errors

There are now 3 errors types.

1. Logical error
2. Unreachable error (e.g. from nom-sql)
3. Unreachable but generated by GlueSQL itself.

Currently I'm using Unreachable prefix for the second case.
And how about 3?
Maybe Conflict prefix?

Apply trait alias to optional store traits

Though some custom storages don't need ALTER TABLE syntaxes, they still need to add it.

impl AlterTable for CustomStorage {}

It looks unnecessary but required.

In execute.rs,

pub fn execute<T: 'static + Debug, U: Store<T> + StoreMut<T> + AlterTable>(
...
...

If trait alias feature ships on stable Rust then, we can improve the code above like this.

#[cfg(feature = "alter-table")]
trait Storage<T: 'static + Debug> = Store<T> + StoreMut<T> + AlterTable>
#[cfg(not(feature = "alter-table"))]
trait Storage<T: 'static + Debug> = Store<T> + StoreMut<T>>

pub fn execute<T: 'static + Debug, U: Storage>(
...
...

It is blocked by this issue: rust-lang/rust#41517
Looking forward to see trait alias in stable Rust.

Change return type of Store trait insert_data

It also requires storage code changes.
one in here, sled_storage.
The others are in gluesql-js : memory-storage and web-storage.

Prior work is done by @ryanhossain9797
#70

Current aggregation functions are case-sensitive

SELECT COUNT(*) FROM TableA;

SELECT count(*) FROM TableA;

both should work but the second one is not working.

Chainging return type of `INSERT`

Current

Payload::Insert(Vec<Row>)

Expected

Payload::Insert(usize)

Let's simply return a number of inserted rows.

Arithmetic operator support

For example,
UPDATE TableA SET v1 = v1 + 1 is not supported because current GlueSQL does not support this arithmetic operator.
Let's make it.

AVG aggregation support

But after the first release

WebAssembly build

After finishing #7
Make WebAssembly build and test using MemoryStorage.

It's required to make JavaScript interface. 🔢

Clarify the use imports in the readme

In the Usage Section it would be better to explicitly put all imports necesary instead of the glob imports as they tend to hurt readability.

CREATE TABLE IF NOT EXISTS support

IF NOT EXISTS syntax is not supported yet.
Let's make it work.

Value::I64(100) == Value::OptI64(Some(100)) but...

assert_eq!(Value::I64(100), Value::OptI64(Some(100)))
// it fails

PartialEq for Value must be implemented manually, but for now, it is applied by using #[derive(PartialEq)]

INSERT with more values is accepted

CREATE TABLE Test (id INTEGER);
INSERT INTO Test VALUES (100, 200);

below insertion query should report an error but it is accepted.

• src/data/row.rs

Current Row struct cares lack of values or columns but not handling more values properly.

MemoryStorage support

Currently the only supported storage is SledStorage.
But it can't be built for Web.

So, make the MemoryStorage and use it for WebAssembly build.

NULLABLE field support

NULLABLE should be supported in the first release.
👍

Allow Nullable keyword in CREATE and apply to Value and join works.

Add Evaluated enum doc comments

#59

enum Evaluated can be quite confusing for newcomers, it's required to have some explanation of this.
Use source code doc comments feature.

Then, someone wants to contributor can see that in source code or by cargo doc --document-private-items.

NULL + 1 should be NULL

#59 (comment)

Current:
NULL + 1 => 1

Expected:
NULL + 1 => NULL

Inserting multiple values

Currently, GlueSQL support only single value insertion for INSERT query.

INSERT INTO TableA VALUES (100, "Foo");

This issue is for supporting multiple value insertion using single INSERT query, like this below:

INSERT INTO TableA VALUES (100, "Hello"), (200, "World"), (300, "GOOOOOD");

Fix `blend.rs` to support Join

1. Current blend.rs does not support Join queries.
   It only takes the first node of BlendContext linked list, but it should handle all properly.

2. And I also need to some kind of macro for helping value comparison test.
   There's an example in basic.rs.

let res = helper.run("select id from test").expect("select");
assert_eq!(res, Payload::Select(vec![Row(vec![Value::I64(1)])]));

Make a macro to shorten and readify this: Payload::Select(vec![Row(vec![Value::I64(1)])]).

Row-unreachable occurs

CREATE TABLE Test (id INTEGER, name INTEGER NULL);

INSERT INTO Test VALUES (100);

Then error occurs on insert query.

{"Row":"Unreachable"}

It is reachable but unreachable!

Add filter.rs integration test cases

related to #56

There doesn't exist place for adding conditional statement tests cases which are used in WHERE clause.
It's obviously better to have, let's add src/tests/filter.rs, like blend.

filter.rs check_expr function is stateless, so we can easily add test cases to reach all possible branches.

It would be good to start from renaming src/tests/between.rs to src/tests/filter.rs.

IS NULL, IS NOT NULL support

SELECT * FROM Foo WHERE name IS NULL;
SELECT * FROM Bar WHERE name IS NOT NULL;

ALTER TABLE support

There exists two store traits: Store and StoreMut.

Plan is adding new Alter store trait, but this time, Alter trait will be optional.

Provide 3 SQL statements

• ADD COLUMN
• DROP COLUMN
• RENAME COLUMN

Arithmetic operator support in blend.rs

SELECT 1 + 2 FROM TableA;

This kind of query above should be supported!

question: is array support planned?

Apologies if you've covered this somewhere already, or if this ought to be asked of sled, but are arrays planned?