GithubHelp home page GithubHelp logo

smallhelm / level-fact-base Goto Github PK

View Code? Open in Web Editor NEW
48.0 5.0 8.0 134 KB

Store immutable facts and query them with datalog.

License: MIT License

JavaScript 100.00%
datomic datalog query-engine immutability ledger

level-fact-base's Introduction

level-fact-base

Test stability - experimental JavaScript Style Guide

Store immutable "facts" in level and query them with datalog.

level-fact-base is inspired by Datomic. Also check out datascript.

Why "Facts"??

A fact is something that happened.

  • E - Entity - an id that represents an entity i.e. "user10"
  • A - Attribute - the attribute the fact is about i.e. "email"
  • V - Value - the attribute value i.e. "some@email"
  • T - Transaction - when this fact became true
  • O - Assertion/Retraction - whether the fact was asserted or retracted

The power of facts

Using a fact based information model inherently provide these 3 benefits.

1 - Flexible data model

When you write data you don't need to think much about how it will be queried later. You simply assert attributes about an entity. Facts are not tightly coupled with structure.

2 - Built in, queryable history and auditing

Most databases only remember the last thing that was true. For example:

 - April 1 -
user10 : Set email to "old@email"

 - April 6 -
user10 : Set email to "new@email"

 - April 18 -
You    : What is user10's email address?
aDumbDB: "new@email"

You    : What was user10's email on April 3?
aDumbDB: "new@email"

But what you really want is this:

You     : What is user10's email address?
FactBase: "new@email", on April 6 it was set by user10

You     : What was user10's email on April 3?
FactBase: "old@email", on April 1 it was set by user10, but on April 6 it was changed to "new@email" by user10

3 - Easy to query with performant joins

Fact base joins are implicit, it simply matches binding variables and unions results. The database is fully indexed for you so you don't need to worry about primary keys or indexes.

For example in SQL:

SELECT
  c.id,
  c.text
FROM
  users u
  JOIN comment c ON c.userId = u.id
WHERE
  u.email = 'my@email'

The fact datalog equivalent:

[
  ['?uid', 'user_email'    , 'my@email'],
  ['?cid', 'comment_userId', '?uid'    ],
  ['?cid', 'comment_text'  , '?text'   ]
]
// implicitly joined on ?uid and ?cid

API

var Transactor = require('level-fact-base')

var db = new Level('db', {
  keyEncoding: require('charwise'),// or bytewise, or any codec for sorted arrays of flat json values
  valueEncoding: 'json'
})

// define a schema for attributes
// like datomic schema is stored and versioned inside the database
var schema = {
  user_name: {type: 'String'},
  user_email: {type: 'String'},
  comment_userId: {type: 'EntityID'}
  comment_text: {type: 'String'}
  // ...
}

var tr = Transactor(db, schema)

// like levelup, every asynchronous function either takes a callback or returns a promise
// i.e. a callback
tr.snap(function(err, fb){ ... })
// or return a Promise
var fb = await tr.snap()

Checkout example/index.js for a more complete example.

tr = Transactor(db, initSchema, nextId = randomUUID)

Initialize the fact-base and return a transactor (tr for short)

  • db is any thing that exposes a levelup api.
  • initSchema the current expected schema for the transactor to use. As part of starting up the transactor it will sync up the schema to match what you pass it.
  • nextId a function to generate unique string ids. Defaults to crypto.randomUUID

tr.snap() -> fb

Asynchronously get the current snapshot of the database.

tr.asOf(txn) -> fb

Asynchronously get a given txn version of the database.

tr.transact(entities) -> fb

Assert facts and get the resulting new version of the database.

transact([
  {
    $e: '101', // the entity id
    email: 'my@email',
    name: 'bob',
    // This expands to:
    // ['101', 'email', 'my@email']
    // ['101', 'name' , 'bob'     ]
  }
], function(err, fb){
  // fb is the new fb version
})

// or
fb = await transact([..])

fb.q(tuples, binding, select) -> results

The main entry point for performing datalog queries. Anything that starts with '?' is a binding variable.

fb.q([[ '?uid', 'user_email'    , '?email' ],
      [ '?cid', 'comment_userId', '?uid'   ],
      [ '?cid', 'comment_text'  , '?text'  ]

      { email: 'my@email' }, // map of bindings i.e. bind ?email to 'my@email'

      [ 'cid', 'text' ], // select which result bindings we care about

      function(err, results){
        // results are
        // [
        //   {cid: '123', text: 'some comment about the post...'},
        //   {cid: '321', text: 'annother comment'},
        //   ...
        // ]
      })

// or
results = await fb.q([..], {..}, [..])

You may also pass filter functions as the values in a binding map. Bound functions should return a boolean and filter out facts that evalutate falsy.

function hasExclamation (text) {
  return text.includes('!')
}

fb.q([[ '?cid', 'comment_userId', '?uid'   ],
      [ '?cid', 'comment_text'  , '?text'  ]

      { text: hasExclamation }, // match comments on text value

      [ 'cid', 'text' ], // select which result bindings we care about

      function(err, results){
        // results are
        // [
        //   {cid: '456', text: 'wow!'},
        //   {cid: '654', text: 'super!'},
        //   ...
        // ]
      })

For more examples see test.js.

NOTE: To help prevent injection attacks, use bindings to pass in untrusted data so it's properly escaped.

fb.get($e) -> entity

A sugar function that simply gets all attributes an entity.

fb.get('101', function(err, user){

  // user is {$e: '101', name: 'bob', email: 'my@email'}

})

// or
user = await fb.get('101')

License

MIT

level-fact-base's People

Contributors

farskipper avatar miguelsm avatar simontegg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

level-fact-base's Issues

A few questions

Very cool library!
I just have a few questions if you don't mind:

  1. Can you point me to where in the source the join magic happens?
  2. The examples show equality-based queries. Are there plans to support other operators like > < etc?

"Error: can only encode arrays" with "Date" type

I get an "Error can only encode arrays" when passing a Date object as a value. When I call .toISOString() on the Date I get the validation error from src/schemaTypes.js

I believe the issue is here https://github.com/dominictarr/charwise/blob/master/codec/object.js#L40

    function encodeItem(item) {
        if (typeof item === 'object') {
            return encode(item);
        }
        return escape(codec.encode(item));
}

charwise just tests for objects, evaluating true for Date then passes it to a function that expects an array.

Stacktrace follows

 node index.js
Error: can only encode arrays
    at encode (/home/simon/projects/lfb/node_modules/charwise/codec/object.js:27:44)
    at encodeItem (/home/simon/projects/lfb/node_modules/charwise/codec/object.js:41:20)
    at Object.encode (/home/simon/projects/lfb/node_modules/charwise/codec/object.js:33:24)
    at Object.exports.encode (/home/simon/projects/lfb/node_modules/charwise/index.js:39:28)
    at Codec.encodeKey (/home/simon/projects/lfb/node_modules/level-codec/index.js:32:45)
    at /home/simon/projects/lfb/node_modules/level-codec/index.js:53:17
    at Array.map (native)
    at Codec.encodeBatch (/home/simon/projects/lfb/node_modules/level-codec/index.js:50:14)
    at DB._batch (/home/simon/projects/lfb/node_modules/encoding-down/index.js:71:20)
    at DB.AbstractLevelDOWN.batch (/home/simon/projects/lfb/node_modules/abstract-leveldown/abstract-leveldown.js:181:8)
    at LevelUP.batch (/home/simon/projects/lfb/node_modules/levelup/lib/levelup.js:255:11)
    at transact (/home/simon/projects/lfb/node_modules/level-fact-base/src/index.js:126:6)
    at /home/simon/projects/lfb/node_modules/level-fact-base/src/index.js:234:7
    at Object.push (/home/simon/projects/lfb/node_modules/fastq/queue.js:88:14)
    at /home/simon/projects/lfb/node_modules/level-fact-base/src/index.js:344:19
    at onLoad (/home/simon/projects/lfb/node_modules/level-fact-base/src/index.js:308:7)
    at Object.transact (/home/simon/projects/lfb/node_modules/level-fact-base/src/index.js:342:7)
    at main (/home/simon/projects/lfb/index.js:85:6)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) undefined

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.