niamtokik / cozo

An interface to cozodb

Founder of Erlang Punch project.

Work on small Erlang code on my spare time.

cozo's Issues

Release this project on hex.pm

Crash during test

cozolib_c is crashing during a test.

thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0', cozo-core/src/fixed_rule/algos/all_pairs_shortest_path.rs:80:24
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Aborted

when executing this test:

?IQUERY_LOG(Db, "centrality[code, score] <~ BetweennessCentrality(*route[a, b])"
  "?[code, desc, score] := centrality[code, score], *airport{code, desc}"
  ":limit 10;"
  ":order -score"),

Create rocksdb engine tests based on official cozodb tutorial

see #3

Add benchmarks

Create mem engine tests based on official cozodb tutorial

A part of the tests are already done, but the tutorial is not done (yet).

Error message with color or UTF8 characters issues

When executing a query, data returned by the interfaces are not correctly parsed

{ok, Db} = cozo:open(), 
(fun({error, E}) -> io:format("~p~n", [E]) end)(cozo:run(Db, "?[] <- [[1,2,3,,]]")), 
cozo:close(Db).
[123,34,99,97,117,115,101,115,34,58,91,93,44,34,99,111,100,101,34,58,34,112,
 97,114,115,101,114,58,58,112,101,115,116,34,44,34,100,105,115,112,108,97,121,
 34,58,34,92,117,48,48,49,98,91,51,49,109,112,97,114,115,101,114,58,58,112,
 101,115,116,92,117,48,48,49,98,91,48,109,92,110,92,110,32,32,92,117,48,48,49,
 98,91,51,49,109,215,92,117,48,48,49,98,91,48,109,32,84,104,101,32,113,117,
 101,114,121,32,112,97,114,115,101,114,32,104,97,115,32,101,110,99,111,117,
 110,116,101,114,101,100,32,117,110,101,120,112,101,99,116,101,100,32,105,110,
 112,117,116,32,47,32,101,110,100,32,111,102,32,105,110,112,117,116,32,97,116,
 32,49,53,46,46,49,53,92,110,32,32,32,9581,9472,9472,9472,9472,92,110,32,92,
 117,48,48,49,98,91,50,109,49,92,117,48,48,49,98,91,48,109,32,9474,32,63,91,
 93,32,60,45,32,91,91,49,44,50,44,51,44,44,93,93,32,92,110,32,32,32,183,32,92,
 117,48,48,49,98,91,51,53,59,49,109,32,32,32,32,32,32,32,32,32,32,32,32,32,32,
 32,9650,92,117,48,48,49,98,91,48,109,92,110,32,32,32,9584,9472,9472,9472,
 9472,92,110,34,44,34,102,105,108,101,110,97,109,101,34,58,34,34,44,34,108,97,
 98,101,108,115,34,58,91,123,34,115,112,97,110,34,58,123,34,108,101,110,103,
 116,104,34,58,48,44,34,111,102,102,115,101,116,34,58,49,53,125,125,93,44,34,
 109,101,115,115,97,103,101,34,58,34,84,104,101,32,113,117,101,114,121,32,112,
 97,114,115,101,114,32,104,97,115,32,101,110,99,111,117,110,116,101,114,101,
 100,32,117,110,101,120,112,101,99,116,101,100,32,105,110,112,117,116,32,47,
 32,101,110,100,32,111,102,32,105,110,112,117,116,32,97,116,32,49,53,46,46,49,
 53,34,44,34,111,107,34,58,102,97,108,115,101,44,34,114,101,108,97,116,101,
 100,34,58,91,93,44,34,115,101,118,101,114,105,116,121,34,58,34,101,114,114,
 111,114,34,125]

Even after an explicit conversion to utf8 binary, the string does not look good.

{ok, Db} = cozo:open(), 
(fun({error, E}) -> io:format("~s~n", [<< <<X/utf8>> || X <- E >>]) end)(cozo:run(Db, "?[] <- [[1,2,3,,]]")), 
cozo:close(Db).
{"causes":[],"code":"parser::pest","display":"\u001b[31mparser::pest\u001b[0m\n\n  \u001b[31mÃ\u001b[0m The query parser has encountered unexpected input / end of input at 15..15\n   âââââ\n \u001b[2m1\u001b[0m â ?[] <- [[1,2,3,,]] \n   · \u001b[35;1m               â²\u001b[0m\n   â°ââââ                       \n","filename":"","labels":[{"span":{"length":0,"offset":15}}],"message":"The query parser has encountered unexpected input / end of input at 15..15","ok":false,"related":[],"severity":"error"}

After filtering, colors code are still present.

Filter = fun(X) when X > 255 -> false; (_) -> true end,
{ok, Db} = cozo:open(), 
(fun({error, E}) -> io:format("~p~n", [lists:filter(Filter, E)]) end)(cozo:run(Db, "?[] <- 
[[1,2,3,,]]")), 
cozo:close(Db).
"{\"causes\":[],\"code\":\"parser::pest\",\"display\":\"\\u001b[31mparser::pest\\u001b[0m\\n\\n  \\u001b[31m×\\u001b[0m The query parser has encountered unexpected input / end of input at 15..15\\n   \\n \\u001b[2m1\\u001b[0m  ?[] <- [[1,2,3,,]] \\n   · \\u001b[35;1m               \\u001b[0m\\n   \\n\",\"filename\":\"\",\"labels\":[{\"span\":{\"length\":0,\"offset\":15}}],\"message\":\"The query parser has encountered unexpected input / end of input at 15..15\",\"ok\":false,\"related\":[],\"severity\":\"error\"}"

It could be due to supports_color crate, to termcolor crate or another configuration somewhere. Even after setting NO_COLOR_TERM=1 environment variable, color tags are still printed.

Create sqlite engine tests based on official cozodb tutorial

Create all tests from cozodb tutorial for sqlite. A cleanup script is also required.

Create Cozoscript parser

based on https://github.com/cozodb/cozo/blob/febf96e4b99fb9d2f579f20780a551c385c372a9/cozo-core/src/cozoscript.pest

Create release files for NIF

The project has been released on hex.pm but the same should be done on Github. It should also be include cozo_nif.so for different architecture, something like that:

• cozo_nif-unknown-linux-x86_64.0.1.0.so

When started, the application should let the user fetch this file.

Improve the output when running a query

At this time, when a query is executed, a long message containing many useless information are provided. A query should return only the full answer and not debugging/benchmark information.

Rethink interfaces and how to deal with queries/responses

The current structure is simple, we generate a query from a string and receive the result as json. A more Erlangish way should be used.

Add support for CozoDB v0.7.5

Find a way to synchronize running databases

CozoDB can use in memory, sqlite and rocksdb databases. Both should be compatible together, and it could be great to create functions to help users to synchronize data between each engine.

Test for memory leaks

Create Continuous Integration pipeline

Create interfaces to deal with distributed queries

Integrating cozodb in Erlang give now a way to deal with distributed queries.

Create a new NIF to support Rust library

The current implementation is using libcozo_c, but this library has lot of limitation. Instead of passing a cozoscript query, we should pass a cozo query data-structure, based on Erlang syntax (or from something close to Erlang syntax).

 When invoking cozoscript:parse/1, this code will

  ____________________       ___________________       ___
 |                    |     |                   |     /   |
 | cozoscript:parse/1 |---->| cozo_rust:parse/1 |--->| {ok, bytecode}
 |____________________|     |___________________|    |____|

 The same bytecode can be reinjected into another module
 lie cozo_bytecode:eval/1 to be evaluated and get back the result.
 
   ___      ______________________      __________________
  /   |    |                      |    |                  |
 |  B |--->| cozo_bytecode:eval/1 |--->| cozo_rust:eval/1 |
 |____|    |______________________|    |__________________|
                                                ||
                                               _||_
                                               \  /
                                                \/
                                               ___
                                              /   |
                                             | {ok, Result}
                                             |____|

 In fact, by using this strategy, we don't modify the current
 structure of cozo application.

                           ______________________
                          |                      |
                     +--->| cozo_nif:run_query/4 |-->Result
  ______________     |    |______________________|
 |              |    |
 | cozo:query/3 |---(+)
 |______________|  / |     ___________________     __________________
                  /  |    |                   |   |                  |
                 /   +--->| cozo_rust:parse/1 |-->| cozo_rust:eval/1 |-->Result
 [based on params]        |___________________|   |__________________|

Cozo Rust Interfaces

A cozo core example can be seen in cozo bin source (REPL and server).

• Cozo core source code, https://github.com/cozodb/cozo/tree/main/cozo-core/src
• Cozo bin source code, https://github.com/cozodb/cozo/tree/main/cozo-bin

Cozoscript to Cozo bytecode

1. send a cozoscript to cozo parser
2. instead of returning a json get the cozo bytecode (or an error)

Erlang Terms to Cozo bytecode

1. identify and understand cozo bytecode generated from cozoscript and based on source code
2. select adequate Erlang Terms to be use as equivalent to cozo bytecode
3. create an erlang terms to cozo bytecode compiler

Cozo bytecode to Erlang Terms

1. based on previous translation, convert cozo bytecode to Erlang terms

Erlang interfaces

1. Create a new interface using Erlang Terms instead of cozoscript

Resources

• Rustler, https://github.com/rusterlium/rustler
• Rustler Documentation, https://docs.rs/rustler/latest/rustler/
• Rustler Example, https://github.com/rusterlium/NifIo

Create note/paper document on the first implementation using libcozo_c

A draft is already present in notes directory. Needs to be upgraded.

Couple of issues starting in Elixir

1. I don't have wget so the installation didn't work on my mac.
2. I worked my way around that problem and tried to use the library in my elixir app.

Note I am a beginner in elixir, so I might be doing something wrong.

When I try to match the output of cozo.open:

{:ok, db} = :cozo.open(:rocksdb, ~c"./dbrocks")

I get the warning:

The pattern can never match the type.

Pattern:
{:ok, _db}

Type:
{:error, _}

So I stumbled through this, printed out the output of cozo.open:

{:ok, {:cozo, 0, :rocksdb, ~c"./dbrocks", %{}, #PID<0.249.0>}}

then I tried to actually run a query, the only thing that worked for me is:

result = :cozo.run(0, String.to_charlist(query))

I don't know what that zero is, but it's the only thing that worked for me.
What I'm trying to do is create a GenServer that starts cozo in init and saves the result in the state.
I tried passing the entire object to cozo.run, but only passing the zero worked.

Am I doing it wrong?

Thanks

convert open/3 to open/1

open/3 function should be called with simple parameter list (or map).

{ok, Db} = cozo:open([{db_engine, mem}, {db_path,"/tmp/database.db"}, {db_options, #{}} })
% or
{ok, Db} = cozo:open(#{ db_engine => mem, db_path => "/tmp/database.db", db_options => #{} })