chiehwen / ejdb

EJDB — Embedded JSON Database engine

Embedded JSON Database engine

It aims to be a fast MongoDB-like library which can be embedded into C/C++ and NodeJS applications under terms of LGPL license.

EJDB is the C library based on modified version of Tokyo Cabinet.

JSON representation of queries and data implemented with API based on C BSON

News

2013-01-22 Collection joins now supported

Features

• LGPL license allows you to embed this library into proprietary software.
• MongoDB-like queries and overall philosophy.
• Collection level write locking.
• Collection level transactions.
• String token matching queries: $stror $strand
• Node.js binding
• Collection joins

Documentation

• EJDB Command line interface
• NodeJS binding
  • Installation
  • Samples
  • NodeJS API
• EJDB C Library
  • Building & Installation
  • Samples
  • C API
• Collection joins

Community

We use EJDB Google group as our mailing list.

NodeJS binding

One snippet intro

var EJDB = require("ejdb");
//Open zoo DB
var jb = EJDB.open("zoo", EJDB.DEFAULT_OPEN_MODE | EJDB.JBOTRUNC);

var parrot1 = {
    "name" : "Grenny",
    "type" : "African Grey",
    "male" : true,
    "age" : 1,
    "birthdate" : new Date(),
    "likes" : ["green color", "night", "toys"],
    "extra1" : null
};
var parrot2 = {
    "name" : "Bounty",
    "type" : "Cockatoo",
    "male" : false,
    "age" : 15,
    "birthdate" : new Date(),
    "likes" : ["sugar cane"]
};

jb.save("parrots", [parrot1, parrot2], function(err, oids) {
    if (err) {
        console.error(err);
        return;
    }
    console.log("Grenny OID: " + parrot1["_id"]);
    console.log("Bounty OID: " + parrot2["_id"]);

    jb.find("parrots",
            {"likes" : "toys"},
            {"$orderby" : {"name" : 1}},
            function(err, cursor, count) {
                if (err) {
                    console.error(err);
                    return;
                }
                console.log("Found " + count + " parrots");
                while (cursor.next()) {
                    console.log(cursor.field("name") + " likes toys!");
                }
                cursor.close(); //It's not mandatory to close cursor explicitly
                jb.close(); //Close the database
            });
});

Installation

System libraries:

• g++
• zlib

On Debian/Ubuntu linux you can install it as follows:

sudo apt-get install g++ zlib1g zlib1g-dev

Installation from node package manager:

npm install ejdb

EJDB NodeJS samples

• node/samples
• node/tests
• nwk-ejdb-address-book

EJDB NodeJS API

### EJDB.open(dbFile, openMode)

Open database. Return database instance handle object.
Default open mode: JBOWRITER | JBOCREAT.
This is blocking function.

Arguments

• {String} dbFile Database main file name
• {Number} [openMode=JBOWRITER | JBOCREAT] Bitmast of open modes: - JBOREADER Open as a reader. - JBOWRITER Open as a writer. - JBOCREAT Create if db file not exists - JBOTRUNC Truncate db.

---

### close()

Close database.
If database was not opened it does nothing.
This is blocking function.

---

### isOpen() Check if database in opened state.

---

### ensureCollection(cname, copts)

Automatically creates new collection if it does't exists. Collection options copts applied only for newly created collection. For existing collections copts takes no effect.

Collection options (copts):

• cachedrecords : Max number of cached records in shared memory segment. Default: 0
• records : Estimated number of records in this collection. Default: 65535.
• large : Specifies that the size of the database can be larger than 2GB. Default: false
• compressed : If true collection records will be compressed with DEFLATE compression. Default: false.

This is blocking function.

Arguments

• {String} cname Name of collection.
• {Object} [copts] Collection options.

---

### dropCollection(cname, prune, cb)

Drop collection.

Call variations:

dropCollection(cname)
dropCollection(cname, cb)
dropCollection(cname, prune, cb)

Arguments

• {String} cname Name of collection.
• {Boolean} [prune=false] If true the collection data will erased from disk.
• {Function} [cb] Callback args: (error)

---

### save(cname, jsarr, cb)

Save/update specified JSON objects in the collection. If collection with cname does not exists it will be created.

Each persistent object has unique identifier (OID) placed in the _id property. If a saved object does not have _id it will be autogenerated. To identify and update object it should contains _id property.

If callback is not provided this function will be synchronous.

Call variations:

save(cname, json object, [cb])
save(cname, <Array of json objects>, [cb])

Arguments

• {String} cname Name of collection.
• {Array|Object} jsarr Signle JSON object or array of JSON objects to save
• {Function} [cb] Callback args: (error, {Array} of OIDs for saved objects)

Return

• {Array} of OIDs of saved objects in synchronous mode otherwise returns {undefined}.

---

### load(cname, oid, cb)

Loads JSON object identified by OID from the collection. If callback is not provided this function will be synchronous.

Arguments

• {String} cname Name of collection
• {String} oid Object identifier (OID)
• {Function} cb Callback args: (error, obj) obj: Retrieved JSON object or NULL if it is not found.

Return

• JSON object or {null} if it is not found in synchronous mode otherwise return {undefined}.

---

### remove(cname, oid, cb)

Removes JSON object from the collection. If callback is not provided this function will be synchronous.

Arguments

• {String} cname Name of collection
• {String} oid Object identifier (OID)
• {Function} cb Callback args: (error)

---

### find(cname, qobj, orarr, hints, cb) Execute query on collection. EJDB queries inspired by MongoDB (mongodb.org) and follows same philosophy.

Supported queries:
  - Simple matching of String OR Number OR Array value:
      -   {'fpath' : 'val', ...}
  - $not Negate operation.
      -   {'fpath' : {'$not' : val}} //Field not equal to val
      -   {'fpath' : {'$not' : {'$begin' : prefix}}} //Field not begins with val
  - $begin String starts with prefix
      -   {'fpath' : {'$begin' : prefix}}
  - $gt, $gte (>, >=) and $lt, $lte for number types:
      -   {'fpath' : {'$gt' : number}, ...}
  - $bt Between for number types:
      -   {'fpath' : {'$bt' : [num1, num2]}}
  - $in String OR Number OR Array val matches to value in specified array:
      -   {'fpath' : {'$in' : [val1, val2, val3]}}
  - $nin - Not IN
  - $strand String tokens OR String array val matches all tokens in specified array:
      -   {'fpath' : {'$strand' : [val1, val2, val3]}}
  - $stror String tokens OR String array val matches any token in specified array:
      -   {'fpath' : {'$stror' : [val1, val2, val3]}}
  - $exists Field existence matching:
      -   {'fpath' : {'$exists' : true|false}}
  - $icase Case insensitive string matching:
      -  {'fpath' : {'$icase' : 'val1'}} //icase matching
      Ignore case matching with '$in' operation:
      -  {'name' : {'$icase' : {'$in' : ['tHéâtre - театр', 'heLLo WorlD']}}}
      For case insensitive matching you can create special type of string index.
  - $elemMatch The $elemMatch operator matches more than one component within an array element.
      -  { array: { $elemMatch: { value1 : 1, value2 : { $gt: 1 } } } }
      Restriction: only one $elemMatch allowed in context of one array field.

  - Queries can be used to update records:

    $set Field set operation.
        - {.., '$set' : {'field1' : val1, 'fieldN' : valN}}
    $upsert Atomic upsert. If matching records are found it will be '$set' operation,
            otherwise new record will be inserted with fields specified by argment object.
       - {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
    $inc Increment operation. Only number types are supported.
        - {.., '$inc' : {'field1' : number, ...,  'field1' : number}
    $dropall In-place record removal operation.
        - {.., '$dropall' : true}
    $addToSet Atomically adds value to the array only if its not in the array already.
                If containing array is missing it will be created.
        - {.., '$addToSet' : {'fpath' : val1, 'fpathN' : valN, ...}}
    $addToSetAll Batch version if $addToSet
        - {.., '$addToSetAll' : {'fpath' : [array of values to add], ...}}
    $pull Atomically removes all occurrences of value from field, if field is an array.
        - {.., '$pull' : {'fpath' : val1, 'fpathN' : valN, ...}}
    $pullAll Batch version of $pull
        - {.., '$pullAll' : {'fpath' : [array of values to remove], ...}}

NOTE: It is better to execute update queries with `$onlycount=true` hint flag
     or use the special `update()` method to avoid unnecessarily rows fetching.

NOTE: Negate operations: $not and $nin not using indexes
      so they can be slow in comparison to other matching operations.

NOTE: Only one index can be used in search query operation.

NOTE: If callback is not provided this function will be synchronous.

QUERY HINTS (specified by `hints` argument):
  - $max Maximum number in the result set
  - $skip Number of skipped results in the result set
  - $orderby Sorting order of query fields.
  - $onlycount true|false If `true` only count of matching records will be returned
                          without placing records in result set.
  - $fields Set subset of fetched fields
       If a field presented in $orderby clause it will be forced to include in resulting records.
       Example:
       hints:    {
                   "$orderby" : { //ORDER BY field1 ASC, field2 DESC
                       "field1" : 1,
                       "field2" : -1
                   },
                   "$fields" : { //SELECT ONLY {_id, field1, field2}
                       "field1" : 1,
                       "field2" : 1
                   }
                 }

Many C API query examples can be found in `tcejdb/testejdb/t2.c` test case.

To traverse selected records cursor object is used:
  - Cursor#next() Move cursor to the next record and returns true if next record exists.
  - Cursor#hasNext() Returns true if cursor can be placed to the next record.
  - Cursor#field(name) Retrieve value of the specified field of the current JSON object record.
  - Cursor#object() Retrieve whole JSON object with all fields.
  - Cursor#reset() Reset cursor to its initial state.
  - Cursor#length Read-only property: Number of records placed into cursor.
  - Cursor#pos Read/Write property: You can set cursor position: 0 <= pos < length
  - Cursor#close() Closes cursor and free cursor resources. Cursor cant be used in closed state.

Call variations of find():
   - find(cname, [cb])
   - find(cname, qobj, [cb])
   - find(cname, qobj, hints, [cb])
   - find(cname, qobj, qobjarr, [cb])
   - find(cname, qobj, qobjarr, hints, [cb])

Arguments

• {String} cname Name of collection
• {Object} qobj Main JSON query object
• {Array} [orarr] Array of additional OR query objects (joined with OR predicate).
• {Object} [hints] JSON object with query hints.
• {Function} cb Callback args: (error, cursor, count) cursor: Cursor object to traverse records qobj count: Total number of selected records

Return

• If callback is provided returns {undefined}
• If no callback and $onlycount hint is set returns count {Number}.
• If no callback and no $onlycount hint returns cursor {Object}.

---

### findOne(cname, qobj, orarr, hints, cb) Same as #find() but retrieves only one matching JSON object.

Call variations of findOne():

findOne(cname, [cb])
findOne(cname, qobj, [cb])
findOne(cname, qobj, hints, [cb])
findOne(cname, qobj, qobjarr, [cb])
findOne(cname, qobj, qobjarr, hints, [cb])

Arguments

• {String} cname Name of collection
• {Object} qobj Main JSON query object
• {Array} [orarr] Array of additional OR query objects (joined with OR predicate).
• {Object} [hints] JSON object with query hints.
• {Function} cb Callback args: (error, obj) obj Retrieved JSON object or NULL if it is not found.

Return

• If callback is provided returns {undefined}
• If no callback is provided returns found {Object} or {null}

---

### update(cname, qobj, orarr, hints, cb) Convenient method to execute update queries.

• $set Field set operation:
  • {some fields for selection, '$set' : {'field1' : {obj}, ..., 'field1' : {obj}}}
• $upsert Atomic upsert. If matching records are found it will be '$set' operation, otherwise new record will be inserted with fields specified by argment object.
  • {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
• $inc Increment operation. Only number types are supported.
  • {some fields for selection, '$inc' : {'field1' : number, ..., 'field1' : {number}}
• $dropall In-place record removal operation.
  • {some fields for selection, '$dropall' : true}
• $addToSet | $addToSetAll Atomically adds value to the array only if its not in the array already. If containing array is missing it will be created.
  • {.., '$addToSet' : {'fpath' : val1, 'fpathN' : valN, ...}}
• $pull | pullAll Atomically removes all occurrences of value from field, if field is an array.
  • {.., '$pull' : {'fpath' : val1, 'fpathN' : valN, ...}}

Call variations of update():

update(cname, [cb])
update(cname, qobj, [cb])
update(cname, qobj, hints, [cb])
update(cname, qobj, qobjarr, [cb])
update(cname, qobj, qobjarr, hints, [cb])

Arguments

• {String} cname Name of collection
• {Object} qobj Update JSON query object
• {Array} [orarr] Array of additional OR query objects (joined with OR predicate).
• {Object} [hints] JSON object with query hints.
• {Function} cb Callback args: (error, count) count The number of updated records.

Return

• If callback is provided returns {undefined}.
• If no callback is provided returns {Number} of updated objects.

---

### count(cname, qobj, orarr, hints, cb) Convenient count(*) operation.

Call variations of count():

count(cname, [cb])
count(cname, qobj, [cb])
count(cname, qobj, hints, [cb])
count(cname, qobj, qobjarr, [cb])
count(cname, qobj, qobjarr, hints, [cb])

Arguments

• {String} cname Name of collection
• {Object} qobj Main JSON query object
• {Array} [orarr] Array of additional OR query objects (joined with OR predicate).
• {Object} [hints] JSON object with query hints.
• {Function} cb Callback args: (error, count) count: Number of matching records.

Return

• If callback is provided returns {undefined}.
• If no callback is provided returns {Number} of matched object.

---

### sync(cb) Synchronize entire EJDB database with disk.

Arguments

• {Function} cb Callback args: (error)

---

### dropIndexes(cname, path, cb) Drop indexes of all types for JSON field path.

Arguments

• {String} cname Name of collection
• {String} path JSON field path
• {Function} [cb] Optional callback function. Callback args: (error)

---

### optimizeIndexes(cname, path, cb) Optimize indexes of all types for JSON field path. Performs B+ tree index file optimization.

Arguments

• {String} cname Name of collection
• {String} path JSON field path
• {Function} [cb] Optional callback function. Callback args: (error)

---

### ensureStringIndex(cname, path, cb) ### ensureIStringIndex(cname, path, cb) ### ensureNumberIndex(cname, path, cb) ### ensureArrayIndex(cname, path, cb)

Ensure index presence of String|Number|Array type for JSON field path. IString is the special type of String index for case insensitive matching.

Arguments

• {String} cname Name of collection
• {String} path JSON field path
• {Function} [cb] Optional callback function. Callback args: (error)

---

### rebuildStringIndex(cname, path, cb) ### rebuildIStringIndex(cname, path, cb) ### rebuildNumberIndex(cname, path, cb) ### rebuildArrayIndex(cname, path, cb)

Rebuild index of String|Number|Array type for JSON field path. IString is the special type of String index for case insensitive matching.

Arguments

• {String} cname Name of collection
• {String} path JSON field path
• {Function} [cb] Optional callback function. Callback args: (error)

---

### dropStringIndex(cname, path, cb) ### dropIStringIndex(cname, path, cb) ### dropNumberIndex(cname, path, cb) ### dropArrayIndex(cname, path, cb)

Drop index of String|Number|Array type for JSON field path. IString is the special type of String index for case insensitive matching.

Arguments

• {String} cname Name of collection
• {String} path JSON field path
• {Function} [cb] Optional callback function. Callback args: (error)

---

EJDB C Library

One snippet intro

#include <tcejdb/ejdb.h>

static EJDB *jb;

int main() {
    jb = ejdbnew();
    if (!ejdbopen(jb, "addressbook", JBOWRITER | JBOCREAT | JBOTRUNC)) {
        return 1;
    }
    //Get or create collection 'contacts'
    EJCOLL *coll = ejdbcreatecoll(jb, "contacts", NULL);

    bson bsrec;
    bson_oid_t oid;

    //Insert one record:
    //JSON: {'name' : 'Bruce', 'phone' : '333-222-333', 'age' : 58}
    bson_init(&bsrec);
    bson_append_string(&bsrec, "name", "Bruce");
    bson_append_string(&bsrec, "phone", "333-222-333");
    bson_append_int(&bsrec, "age", 58);
    bson_finish(&bsrec);
    //Save BSON
    ejdbsavebson(coll, &bsrec, &oid);
    fprintf(stderr, "\nSaved Bruce");
    bson_destroy(&bsrec);

    //Now execute query
    //QUERY: {'name' : {'$begin' : 'Bru'}} //Name starts with 'Bru' string
    bson bq1;
    bson_init_as_query(&bq1);
    bson_append_start_object(&bq1, "name");
    bson_append_string(&bq1, "$begin", "Bru");
    bson_append_finish_object(&bq1);
    bson_finish(&bq1);

    EJQ *q1 = ejdbcreatequery(jb, &bq1, NULL, 0, NULL);

    uint32_t count;
    TCLIST *res = ejdbqrysearch(coll, q1, &count, 0, NULL);
    fprintf(stderr, "\n\nRecords found: %d\n", count);

    //Now print the result set records
    for (int i = 0; i < TCLISTNUM(res); ++i) {
        void *bsdata = TCLISTVALPTR(res, i);
        bson_print_raw(stderr, bsdata, 0);
    }
    fprintf(stderr, "\n");

    //Dispose result set
    tclistdel(res);

    //Dispose query
    ejdbquerydel(q1);
    bson_destroy(&bq1);

    //Close database
    ejdbclose(jb);
    ejdbdel(jb);
    return 0;
}

You can save this code in csnippet.c And build:

gcc -std=c99 -Wall -pedantic  -c -o csnippet.o csnippet.c
gcc -std=c99 -Wall -pedantic  -o csnippet csnippet.o  -ltcejdb

Building & Installation

Prerequisites

System libraries:

• gcc
• zlib

On Debian/Ubuntu linux you can install it as follows:

   sudo apt-get install gcc zlib1g zlib1g-dev

Building

   cd ./tcejdb
   ./configure --prefix=<installation prefix> && make && make check
   make install

• library name: tcejdb (with pkgconfig)
• main include header: <tcejdb/ejdb.h>

C API

EJDB API presented in ejdb.h C header file.

JSON processing API: bson.h

Queries

/**
 * Create query object.
 * Sucessfully created queries must be destroyed with ejdbquerydel().
 *
 * EJDB queries inspired by MongoDB (mongodb.org) and follows same philosophy.
 *
 *  - Supported queries:
 *      - Simple matching of String OR Number OR Array value:
 *          -   {'fpath' : 'val', ...}
 *      - $not Negate operation.
 *          -   {'fpath' : {'$not' : val}} //Field not equal to val
 *          -   {'fpath' : {'$not' : {'$begin' : prefix}}} //Field not begins with val
 *      - $begin String starts with prefix
 *          -   {'fpath' : {'$begin' : prefix}}
 *      - $gt, $gte (>, >=) and $lt, $lte for number types:
 *          -   {'fpath' : {'$gt' : number}, ...}
 *      - $bt Between for number types:
 *          -   {'fpath' : {'$bt' : [num1, num2]}}
 *      - $in String OR Number OR Array val matches to value in specified array:
 *          -   {'fpath' : {'$in' : [val1, val2, val3]}}
 *      - $nin - Not IN
 *      - $strand String tokens OR String array val matches all tokens in specified array:
 *          -   {'fpath' : {'$strand' : [val1, val2, val3]}}
 *      - $stror String tokens OR String array val matches any token in specified array:
 *          -   {'fpath' : {'$stror' : [val1, val2, val3]}}
 *      - $exists Field existence matching:
 *          -   {'fpath' : {'$exists' : true|false}}
 *      - $icase Case insensitive string matching:
 *          -    {'fpath' : {'$icase' : 'val1'}} //icase matching
 *          Ignore case matching with '$in' operation:
 *          -    {'name' : {'$icase' : {'$in' : ['tHéâtre - театр', 'heLLo WorlD']}}}
 *          For case insensitive matching you can create special index of type: `JBIDXISTR`
 *     - $elemMatch The $elemMatch operator matches more than one component within an array element.
 *          -  { array: { $elemMatch: { value1 : 1, value2 : { $gt: 1 } } } }
 *          Restriction: only one $elemMatch allowed in context of one array field.
 *
 *  - Queries can be used to update records:
 *       $set Field set operation.
 *           - {.., '$set' : {'field1' : val1, 'fieldN' : valN}}
 *       $upsert Atomic upsert. If matching records are found it will be '$set' operation,
 *              otherwise new record will be inserted with fields specified by argment object.
 *           - {.., '$upsert' : {'field1' : val1, 'fieldN' : valN}}
 *       $inc Increment operation. Only number types are supported.
 *           - {.., '$inc' : {'field1' : number, ...,  'field1' : number}
 *       $dropall In-place record removal operation.
 *           - {.., '$dropall' : true}
 *       $addToSet Atomically adds value to the array only if its not in the array already.
 *                    If containing array is missing it will be created.
 *           - {.., '$addToSet' : {'fpath' : val1, 'fpathN' : valN, ...}}
 *       $addToSetAll Batch version if $addToSet
 *           - {.., '$addToSetAll' : {'fpath' : [array of values to add], ...}}
 *       $pull Atomically removes all occurrences of value from field, if field is an array.
 *           - {.., '$pull' : {'fpath' : val1, 'fpathN' : valN, ...}}
 *       $pullAll Batch version of $pull
 *           - {.., '$pullAll' : {'fpath' : [array of values to remove], ...}}
 *
 *  NOTE: Negate operations: $not and $nin not using indexes
 *  so they can be slow in comparison to other matching operations.
 *
 *  NOTE: Only one index can be used in search query operation.
 *
 *  QUERY HINTS (specified by `hints` argument):
 *      - $max Maximum number in the result set
 *      - $skip Number of skipped results in the result set
 *      - $orderby Sorting order of query fields.
 *      - $fields Set subset of fetched fields
            If a field presented in $orderby clause it will be forced to include in resulting records.
 *          Example:
 *          hints:    {
 *                      "$orderby" : { //ORDER BY field1 ASC, field2 DESC
 *                          "field1" : 1,
 *                          "field2" : -1
 *                      },
 *                      "$fields" : { //SELECT ONLY {_id, field1, field2}
 *                          "field1" : 1,
 *                          "field2" : 1
 *                      }
 *                    }
 *
 * Many query examples can be found in `testejdb/t2.c` test case.
 *
 * @param EJDB database handle.
 * @param qobj Main BSON query object.
 * @param orqobjs Array of additional OR query objects (joined with OR predicate).
 * @param orqobjsnum Number of OR query objects.
 * @param hints BSON object with query hints.
 * @return On success return query handle. On error returns NULL.
 */
EJDB_EXPORT EJQ* ejdbcreatequery(EJDB *jb, bson *qobj, bson *orqobjs, int orqobjsnum, bson *hints);

EJDB C Samples

You can find some code samples in:

• tcejdb/samples
• tcejdb/testejdb

Basic EJDB architecture

EJDB database files structure

.
├── <dbname>
├── <dbname>_<collection1>
├── ...
├── <dbname>_<collectionN>
└── <dbname>_<collectionN>_<fieldpath>.<index ext>

Where

• <dbname> - name of database. It is metadata DB.
• <collectionN> - name of collection. Collection database.
• <fieldpath> - JSON field path used in index
• <index ext> - Collection index extension:
  • .lex String index
  • .dec Number index
  • .tok Array index

Limitations

• One ejdb database can handle up to 1024 collections.
• Indexes for objects in nested arrays currently not supported (#37)

TODO

• Collect collection index statistic
• Windows port

Related software

Connect session store backed by EJDB database