dbcaml / dbcaml

DBCaml is a database library for OCaml

A database toolkit built on Riot

Quick Start | Documentation | Examples |   

DBCaml is an async database toolkit built on Riot, an actor-model multi-core scheduler for OCaml 5. DBCaml is inspired by Elixirs where the developer can spin up a connection manager and connections the manager takes cares of.

** Note: DBCaml is currently in development and is not ready for production. Only for testing purposes **

type user = {
  name: string;
  id: int;
  some_int64: int64;
  some_int32: int32;
  some_float: float;
  some_bool: bool;
  pet_name: string option;
  pets: string list;
  pets_array: string array;
}
[@@deriving deserialize]

type users = user list [@@deriving deserialize]

(* Start the database connection pool *)
let* db =
  let config =
    Silo.config
      ~connections:5
      ~connection_string:
        "postgresql://postgres:postgres@localhost:6432/postgres?sslmode=disable"
  in

  match Silo.connect ~config with
  | Ok c -> Ok c
  | Error (`Msg e) -> Error e
in

(* Fetch the user and return the user to a variable *)
let* fetched_users =
  Silo.query
    db
    ~query:
      "select name, id, some_bool, pet_name, some_int64, some_int32, some_float, pets, pets as pets_array from users limit 2"
    ~deserializer:deserialize_users
in

List.iter
  (fun x ->
    Printf.printf
      "Fetching user with id %d:\nName: %s\nSome float: %f\nSome int64: %d\nSome int32: %d\n%s\n Some bool: %b\nPets: %s\nPets array: %s\n\n"
      x.id
      x.name
      x.some_float
      (Int64.to_int x.some_int64)
      (Int32.to_int x.some_int32)
      (match x.pet_name with
      | Some pn -> Printf.sprintf "Pet name: %S" pn
      | None -> "No pet")
      x.some_bool
      (String.concat ", " x.pets)
      (String.concat ", " (Array.to_list x.pets_array)))
  (Option.get fetched_users);

DBCaml aims to offer:

• Database pooling. Built using Riots lightweight process to spin up a connection pool.

• Database Agnostic. Support for Postgres, and more to come (MySQL, MariaDB, SQLite).

• Built in security. With built-in security it allows users to focus on writing queries, without being afraid of security breaches.

• Cross Platform. DBCaml compiles anywhere!

• Not an ORM. DBCaml is not an orm, it simply handles the boring stuff you don't want to deal with and allows you to have full insight on what's going on.

Quick Start

opam pin dbcaml.0.0.2 git+https://github.com/dbcaml/dbcaml

After that, you can use any of the examples as a base for your app, and run them:

dune exec X

Important

DBCaml is heavily in development, the content in this repo will change. It's not production ready and will probably have bugs

dbcaml's Issues

Security

DBCaml need help with preventing security bugs. Some of this are:

• SQL injections
• TLS

SQLite Driver

We need a SQLite Driver. The idea is that each of this drivers implements Driver.t type just as we are doing with https://github.com/dbcaml/dbcaml/blob/main/dbcaml_driver_postgres/dbcaml_driver_postgres.ml

Move to bytestring instead of buffers for drivers

We should use Bytestring instead of Buffer for handling the data we work with as it's more efficient and don't allocate more memory.

Write tests for the libraries

As we start to know what path this library will make, is it possible to write tests.
We should write integration tests to verify that stuff works the way we expect them to do.

Fetch functions and mapping the data to types

Atm do each driver map it's result to string option list and the idea is that we later on should be able to take a record and map fields to the fields. This is the idea of Row.ml file.

Something we also need to think about when we talk about this is that if string option is the "correct" type or if we should rethink into returning something else and how could this if so be mapped to a type?

I did create a tiny function called map_to which idea was to take string option listand map it, however, I am not done with it yet.
The function looks like this:

let map_to (row : t) =
  List.fold_left
    (fun acc s ->
      match s with
      | Some v -> v :: acc
      | None -> acc)
    []
    row

TODO:

• Map result to a type which each fetch function use
• Implement more fetch functions
• Implement a cursors for fetch_many

PoolManager does not currently start up the connections, it just store a already established connection

Currently do we store a already established connection in the connection pool instead of taking a function that startup the connection in a child_spec. This is a issue due to that if the connection is killed and the process dies won't the SuperVisor be able to start the connection again as the connection is dead and the holder holds a connection.

To solve this should the holder take a function that starts the connection, not a already existing connection

Tests

We need to add tests and this is some of the tests that I thought of:

• To test that each driver works against multiple databases
• Test that the connection pool works the way we expect it to
• Mapping to records

Create issue templates

We need to create issue templates to make it easier to report issues

MySQL & MariaDB Driver

We need MySQL and MariaDB Drivers. The idea is that each of this drivers implements Driver.t type just as we are doing with https://github.com/dbcaml/dbcaml/blob/main/dbcaml_driver_postgres/dbcaml_driver_postgres.ml

Fix actions

Seems as if the actions are failing due to being unable to access postgres database.

Connection pool

Currently do we only work 1 connection and nothing more and we are not using Riot as we want to. The idea is to use Riot to create a connection manager(CM) + connections where the connection pool are aware of which CM that is currently in use and not. This way can the CM distribute queries to the database using the connection pool.

As we have a type called Driver.t and Connection.t is the idea that the CM will use this type to send queries without being aware of which driver it is working with.

The arguments for the connection could be Driver.t and max_connections. The Driver.t represents the driver and max_connections is the amount of connections to create. ´

Add CONTRIBUTING

We need to add a CONTRIBUTING file to help people who wants to contribute to get up and running within the code.

Petrol like query builder library

Petrol is a SQL library we should take inspiration from on this task. The idea is that instead of writing string queries should you be able to use types to make queries.
Petrol have a pretty good interface for this:

open Petrol
open Petrol.Sqlite3

(* define a new schema *)
let schema = StaticSchema.init ()

(* declare a table *)
let example_table, Expr.[name; age] =
    StaticSchema.declare_table schema ~name:"example"
    Schema.[
        field "name" ~ty:Type.text;
        field "age" ~ty:Type.int
    ]
    
    (* create a query *)
let insert_person ~name:n ~age:a db =
    Query.insert ~table:example_table
        ~values:Expr.[
            name := s n;
            age := i a
         ]
    |> Request.make_zero
    |> Petrol.exec db

Generate types of the sql query during build time.

During build time could we read the sql query (select id, users from users) and create a type based on this query automatically during build time using preprocessing(ppx).

The only thing we need to consider is that we need to have a db connection to make this work as we need to know what type the column have in the database.

Branch out the pool to it's own library

To allow more developers to use the pool in DBCaml should we make the pool it's own library which DBCaml uses to operate.

The documentation for this library should exist at dbca.ml

Postgres Driver to have it owns socket

Instead of using postgresql should we move out to write our own socket for postgres. This could make it easier to integrate it with Riot and prevent the usage of libsql-dev

DBCaml pool sometimes never get to the point of being ready for query

Sometimes when we start dbcaml do we get a infinitie loop where we never get to the point of being able to start querying.

Also when we start the pool do we get a lot of logs saying

11:55:28.51995Z [dbcaml.connection_manager::debug] not enough holders, retrying...
11:55:28.51995Z [dbcaml.connection_manager::debug] Acquire a holder lock for requester: <0.1.0>

Branch out the query methods into it's own library

The idea is to split DBCaml into multiple libraries and 1 of the libraries we need to create is the highest level package which will take care of :
1 - Makeing queries towards DBcaml(calling the execute funciton)
2 - Mapping data to types

This change also mean that we need to change what methods DBCaml have so it only have 1 method which is used. The type of the data could be cursors/streaming data/bytes and we need to look into what's best for this.

This new library should handle:

• Map response from DBCaml execute function to types
• fetch functions(fetch_one, fetch_many and so on)
• Params types(numbers, strings, floats, bools and so on)
• SSL upgrade if sslmode=disable isn't provided in the connection string
  Sanitize parama

Developing feedback

1 of the goals with DBCaml is to help the user get feedback while developing and this could be done using macros which executes during build time.
I think SQLX handles this really good and we should take inspiration from SQLX

Fix CI tests

Currently do not our CI pipeline work due to that the CI is not able to install all the packages.

Solution: Opam should get the pin versions for the failing github links in the .opam files which could be a solution.

Add Turso driver

We want to add Turso driver for DBCaml

[DBCaml] test issue

** Describe the bug **
Describe the bug and what error you get

** Reproduction **
Please provide information on how to reproduce this. Links and images are welcome.

** Expected output **

Documentation

We need to create documentation for DBcaml that we can publish somewhere. The idea is to use the built in function to generate doc as html and publish.

Publish to OPAM

We need to publish DBCaml to Opam

Testing tools

DBCaml should be able to help with testing for developers applications and some ideas I have in mind are

• Write a driver that we can use for testing, this driver should also be used to test this library.
• Tools that can be used to verify that data is correct
• Be able to mock response for the fetch functions

Connection Pool.

the idea of the connection pool within DBCaml is that the connection pool should have x connections and a manager for the pool.

The manager should be responsible for:

1. Knowing which connections that is available

THIS is a WIP and will be updated

Contributing

To help more developers which are interested to work with DBCaml should we write a contributing.md with information on how they can contribute. Ex get a local database up and running and also store some information such as connection strings.