rethinkdb / horizon

Horizon is a realtime, open-source backend for JavaScript apps.

Horizon

Official Repository

What is Horizon?

Horizon is an open-source developer platform for building sophisticated realtime apps. It provides a complete backend that makes it dramatically simpler to build, deploy, manage, and scale engaging JavaScript web and mobile apps. Horizon is extensible, integrates with the Node.js stack, and allows building modern, arbitrarily complex applications.

Horizon is built on top of RethinkDB and consists of four components:

• Horizon server -- a middleware server that connects to/is built on top of RethinkDB, and exposes a simple API/protocol to front-end applications.
• Horizon client library -- a JavaScript client library that wraps Horizon server's protocol in a convenient API for front-end developers.
• Horizon CLI - hz -- a command-line tool aiding in scaffolding, development, and deployment
• GraphQL support -- the server will have a GraphQL adapter so anyone can get started building React/Relay apps without writing any backend code at the beginning. This will not ship in v1, but we'll follow up with a GraphQL adapter quickly after launch.

Horizon currently has all the following services available to developers:

• ✅ Subscribe -- a streaming API for building realtime apps directly from the browser without writing any backend code.
• ✅ Auth -- an authentication API that connects to common auth providers (e.g. Facebook, Google, GitHub).
• ✅ Identity -- an API for listing and manipulating user accounts.
• ✅ Permissions -- a security model that allows the developer to protect data from unauthorized access.

Upcoming versions of Horizon will likely expose the following additional services:

• Session management -- manage browser session and session information.
• Geolocation -- an API that makes it very easy to build location-aware apps.
• Presence -- an API for detecting presence information for a given user and sharing it with others.
• Plugins -- a system for extending Horizon with user-defined services in a consistent, discoverable way.
• Backend -- an API/protocol to integrate custom backend code with Horizon server/client-libraries.

Why Horizon?

While technologies like RethinkDB and WebSocket make it possible to build engaging realtime apps, empirically there is still too much friction for most developers. Building realtime apps now requires understanding and manually orchestrating multiple systems across the software stack, understanding distributed stream processing, and learning how to deploy and scale realtime systems. The learning curve is quite steep, and most of the initial work involves boilerplate code that is far removed from the primary task of building a realtime app.

Horizon sets out to solve this problem. Developers can start building apps using their favorite front-end framework using Horizon's APIs without having to write any backend code.

Since Horizon stores data in RethinkDB, once the app gets sufficiently complex to need custom business logic on the backend, developers can incrementally add backend code at any time in the development cycle of their app.

Get Involved

We'd love for you to help us build Horizon. If you'd like to be a contributor, check out our Contributing guide.

Also, to stay up-to-date on all Horizon related news and the community you should definitely join us on Slack or follow us on Twitter.

FAQ

Check out our FAQ at horizon.io/faq

How will Horizon be licensed?

The Horizon server, client and cli are available under the MIT license

horizon's Issues

Server: use proper logging utility

Instead of using console.* methods everywhere, it probably makes sense to use a richer logging utility like winston. The main advantage being that it is transport-agnostic. With little/no additional effort, logs could be piped to console, syslog, websockets (to notify Fusion clients of warnings per #14), or even a RethinkDB table.

If you guys think it makes sense, I'd be happy to work on this. It's mostly busy work and I'd like an excuse to dig into the code anyway.

Add attempted operation with error messages?

Unhandled promise rejection Error: The document with id "8hguf95y5kkuu" was missing.

Currently trying to debug the todo app and I'm pretty sure I know why I'm getting this error but I think it might be more helpful if the error message reported what operation was attempted on the document with that id. Would this be possible?

And then it would also help to tell which connected client was the one requesting the operation?

custom, not database action requiring events

Say I wanted to implement the "user so and so is typing" feature. I could update or insert a document and use subscribe() to get that info. I wouldn't need that "user was typing information" for anything and so updating/inserting would actually be waste of I/O. Having a mechanism for passing just that kind of information might be useful. Tho if RethinkDB implements memory tables, that could be used for this purpose nicely, I think.

Server: switch to middleware-based stack

Related to #12, discussion continued from #21. I think there are three options here:

1. Build fusion server on top of an existing framework (i.e. hapi, koa, or express)
2. Split out core functionality of fusion and then provide middleware for various frameworks
3. Using one of the integrations from (2), additionally provide a standalone server product

Option 1:

I think the main advantage with this option is that we retain more control and potentially have a more rigid API surface for third-party plugins. The downside is that, as a standalone server, this would be difficult to integrate with existing Node apps. Furthermore, if the idea is to have users flesh out their API on top of fusion, everyone is probably going to wish we'd picked the framework they're already most familiar with.

However, we could potentially abstract away the differences between server middleware and fusion plugins into a single unified API.

Option 2:

This seems to be the most flexible option. Even if we only roll out official middleware for one of the frameworks, people could write middleware for their favorite framework based on fusion-core. It also integrates well with existing Node apps and doing so would look something like:

// express example
var fusion = require('express-fusion');
app.use(fusion());

The big question here is how to do plugins. Any non fusion-specific functionality like authentication would just be framework-specific middleware, which would also be true using option (1). The obvious answer is for the fusion middleware (in this case express-fusion) to accept plugin configuration that gets passed down to fusion-core. There are a bunch of ways we could do it, but users will have to understand the difference and directly interact with both fusion plugins and middleware separately.

Option 3:

Possibly bridges the gap a bit between (1) and (2). We get all the benefits of (2) in that you can still bring your own Node server, but we still basically provide solution (1) for people who don't want to deal with any of that.

Allow for .subscribe() to prevent returning initial results

I'm figuring out "Fusion" by ducktaping an app that uses jquery fullcalendar. ( it's here: https://rdb.space/cal/ ) While it's not actual app that's supposed to do anything worth using, I've come up with a problem with subscribe returning the initial results.

Problem is, that fullcalendar uses a function to fetch the events for the calendar

this.eventSource = function(start, end, timezone, callback){
        // can't currently use "between() type of functionality with fusion if you filter by Date object
        //this.backEndCal.above({start: new Date(start)}).below({start: new Date(end)})
        this.backEndCal.findAll({calendarId: "test"}).value().then(function(results){
            console.log(results);
            callback(results);
        });
    }

It was suggested to me, that I use the initial value from subscribe() to populate the calendar, but I don't want to do that, because fullcalendar is capable of lazy loading the events and I feel I wouldn't want to have the actual app load every event, just month or two.

What happens when I use the eventSource and subscribe() is, every event gets rendered twice:

If it's not possible to prevent getting the initial results from subscribe, I think it's at least not totally obvious that every initial result fires .onAdded()

Add type tags to changefeeds in the fusion server

Per discussion in rethinkdb/rethinkdb#5188, it should be easy to insert these tags into the fusion changefeeds on the server side until we get them from the database itself.

Right now the testing framework is inserting them, and there are a couple more places it would be useful to have them

Serve fusion.js file with gzip/bzip

See #22 which was closed when we switched to next

.replace() works once, then never again.

This is my function for handling toggles on a Todo. Utils.extend is basically ReQL's .merge and this.todosDB is my Fusion collection.

app.TodoModel.prototype.toggle = function (todoToToggle) {
    console.log(todoToToggle);
    this.todosDB.replace(
        Utils.extend({}, todoToToggle, {completed: !todoToToggle.completed}) 
    );
};

And here is where I'm defining my changefeed handlers.

app.TodoModel.prototype.subscribeChangefeeds = function(){
        this.todosDB.subscribe({
            onAdded: (added) => {
                this.todos = this.todos.concat(added);
                this.inform();
            },
          onChanged: (changed) => {
                console.log("CHANGED");
                console.log(changed);
                this.todos = this.todos.map((todo) => {
                    return todo.id !== changed.id ? todo : Utils.extend({}, todo, changed);
                });
                this.inform();
            },
            onRemoved: (removed) => {
                this.todos = this.todos.filter((todo) => {
                    return todo.id !== removed.id;
                });
                this.inform();
            }
        });
    };

Every time I click the toggle button this function runs and the console.log output is seen in the console. But I only receive a changefeed event "onChanged" for the first time I click the button and won't work again until restarting. So the console.log in the toggle function hits every time yet the onChanged does not.

Dev mode

We've discussed a couple of ideas about what dev mode should do, and how to discourage people from using it on their production servers.

Dev mode benefits:

1. auto index creation
2. auto table creation
3. (maybe) insecure websockets

Possible dev mode limitations

• limited number of simultaneous client connections

This looks like trial-ware but the idea is to have limitations which you're unlikely to trip while prototyping, but you'll trip very quickly when doing load tests etc.

Passportjs support for Authentication

Passportjs is now standard as it allows authentication from hundreds of mobile devices.
It seems like a separate auth thread isn't considering it as a possibility.

We should explore Loopback auth module and iOS/Android SDKs which support it as well.
https://github.com/strongloop/loopback-component-passport

Protocol changes to accept the "store matrix"

Right now, the client is doing:

• missing: error, conflict: update -> update
• missing: error, conflict: replace -> replace
• missing: insert, conflict: error -> insert
• missing: insert, conflict: update -> upsert
• missing: insert, conflict: replace -> store

The issue is that the protocol doesn't match this very well, it's hard to match up queries with protocol requests. So I think we should add back in update, replace, insert, upsert to the type parameter, and remove the missing and conflict options.

ping @Tryneus

Optimistic updates

I think in some instances the user will want to set the id of a document before saving it to RethinkDB.

This will help in situations where saving a brand new document to the database (versus replacing/updating) and I've also setup a subscription to modifications to the collection. Without having an id, I'll get a document through the subscription once it's been saved to RethinkDB and then I'll have to make sure to update that document once I know the id. Basically, this prevents having to account for the client state of the original document not having an id and the document returned via changefeed with the id field added.

By setting the id client side I can then do a check on all documents incoming to me via subscriptions to see whether I have or have not any version of state of that document currently and then react accordingly.

One way to do this is just add fusion.uuid() to the client lib:

const todos = fusion("todos");

thingTodo = { 
    task: "babelify everything",
    completed: false,
    id: fusion.uuid(),
}

todos.store(thingTodo); 

// Handle incoming changefeeds by keeping a Map or some other structure to easily/efficiently 
//  check for membership of documents I already have. 

An alternative would be to automatically add an id to documents that don't have one via setting up an automatic callback when the result is returned from the Fusion server:

const todos = fusion("todos");

thingTodo = { 
 task: "babelify everything",
 completed: false,
}

todos.store(thingTodo); 

// Immediately

console.log(thingTodo);
// { 
//    task: "babelify everything",
//    completed: false,
//    id: undefined,
// }

// Later on...

console.log(thingTodo);
// { 
//    task: "babelify everything",
//    completed: false,
//    id: "actual_uuid_string",
// }

The other other way to kind of solve this is to encourage subscriptions on single documents versus entire collections, but you would still have the problem of dealing with additions made from other connections to the Fusion server.

Spec out and build a presence API (i.e. is this user logged in?)

TBD

Android / iOS clients

This is a follow-up for rethinkdb/rethinkdb#5239 (comment) .

Besides our JavaScript Fusion "driver", we should look into opening Fusion up for mobile app developers.

I'm not sure what exactly this will take, but @Rajan mentions some possible paths in rethinkdb/rethinkdb#5239 (comment) .

Also related to #58

Unhandled rejection Error: not opened at WebSocket.send

Got this error message on the Fusion server today. Not sure how to reproduce it (I was running various tests in various ways and noticed the error much later).

Unhandled rejection Error: not opened
    at WebSocket.send (/Users/coffeemug/projects/fusion/server/node_modules/ws/lib/WebSocket.js:218:16)
    at Client.send_response (/Users/coffeemug/projects/fusion/server/src/client.js:105:17)
    at Client.handle_response (/Users/coffeemug/projects/fusion/server/src/client.js:98:12)
    at /Users/coffeemug/projects/fusion/server/src/client.js:79:47
    at tryCatcher (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/promise.js:507:31)
    at Promise._settlePromiseAt (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/promise.js:581:18)
    at Promise._settlePromises (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/promise.js:697:14)
    at Async._drainQueue (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/Users/coffeemug/projects/fusion/server/node_modules/bluebird/js/main/async.js:15:14)
    at processImmediate [as _immediateCallback] (timers.js:371:17)

static file api

It would be nice to be able to upload static files for the fusion server to serve. Probably with the admin interface/configuration api.

Spec out and build an authentication API

TBD

Managing reconnection

@Tryneus brought up a question of what to do if there are disconnections. There are two types of disconnects:

• The Fusion client (browser) disconnects from the Fusion server
• The Fusion server disconnects from RethinkDB

How do we handle that?

My initial impulse is that in either of those cases, the client-side app should be made aware of disconnections/reconnections, but all event setup should be automatically maintained/recreated if possible (otherwise we're pushing unnecessary boilerplate version on the user).

So:

var ref = new Fusion('localhost');
ref.on('connected', ()=>...) 
   .on('disconnected', ()=>...);

However, any feeds the client has created should be maintained and should pick up where they left off. This may be very challenging technically until we add restartable feeds (and even after that, it'll be very hard until we add full-blown restartable feeds), so for now we might want to also issue an "emergency" event in case the feeds have been lost irrecoverably:

// Find better name than `all_is_lost`
var ref = new Fusion('localhost');
ref.on('all_is_lost', ()=>{refresh the whole page});

Find a name better than `Fusion`

A few people pointed out that "Fusion" isn't a great name -- there are already lots of things named "Fusion", it doesn't work too well with RethinkDB's brand, etc.

I don't want to debate the name instead of actually building, so I'm locking this issue for now, but once we get enough up and running, I'll reopen this issue so we can think of a better name.

Add automated certificate registration with LetsEncrypt api

Firebase sets up a cert for you to streamline the process when you sign up with them, we could easily do the same thing: https://github.com/letsencrypt/letsencrypt

Get something basic working

In order for us to start building various APIs in Fusion, parallelizing the work, defining the server protocol, etc. we should get a very basic version of the server and the client library up and running.

I'm going to describe a minimal client library API below that will allow connecting to the Fusion server, inserting data, and setting up a stream. For the moment, let's ignore the question of what the server protocol will look like -- once we get a basic server up and running that handles some basic commands, we can go back and define a proper server-side protocol. The following is a basic client-library API the user would use in the browser.

Connecting and getting a reference to a table/collection:

// Connect to the Fusion server
var ref = new Fusion('http://localhost:8181');

// Access a specific collection (which translates to a RethinkDB table in the `test` database).
// If the table doesn't exist, Fusion server will automatically create it.
var items = ref('todo-items');

Listening on events:

// Listen on events on a table. Returns an EventEmitter.
items.on('added', item => {
  // `item` was added to the db
}).on('removed', item => {
  // `item` was removed from the db
});

Inserting data:

// Insert a json object. Returns a promise.
items.insert({ item: 'Get milk.' }).then((id) => {
  // Item inserted successfully, with generated `id`
}).catch(err => {
  // Something bad happened
});

Once we get the basic client library/server infrastructure going for this, it'll be much easier to add more commands. I'll start speccing out the API in the meantime, but in general I think we should support the following operations:

• Reads: get, getAll, between, orderBy (indexed only), limit
• Writes: insert, delete, update, replace

(We'll probably rename/consolidate these so people don't get confused when they start using ReQL)

Errors connecting to websocket / tests

Hey all, just getting started with Fusion. Trying to get it set up and having some trouble, when I run the tests I get this:

I've also tried mimicing the test setup on my own running a fresh rethinkdb and the server, and including this in my head:

And then running:

const fusion = new Fusion('localhost:8181', { secure: false, debug: true })
let chats

fusion.on('connected', () => {
    chats = fusion('chats')
    console.log('were connected')
}))

I get this CLOSING state error, and when I try and do chats.value() the promise doesn't resolve. Using master as of today. Definitely doing something weird on my end.

Also general question, the fusion() call, that connects to the table? Where do you set the database? Apologies for general lack of expertise here. Also final question, is there a chat room for fusion?

Thanks all!

Offline support

We should think about offline support at some point. This was mentioned by @Rajan in rethinkdb/rethinkdb#5239 (comment) .

The idea is that it will make it a lot easier to build mobile applications on top of Fusion.

As a first simpler step, we could allow mirroring certain data to the client for read-only access.
In a second step, we could hook into our optimistic write infrastructure (that we still need to build, #42) to allow for syncing back writes once the user gets online again.

Spec out and build a permissions API/security model

TBD

Spec out and build a geolocation API

TBD

Server: share feeds between connected clients

This would be solved more elegantly with persistent changefeeds in core, but until then I think Fusion needs to handle it transparently.

Rationale

Users will expect similar performance out of Fusion as they would get by rolling their own websocket wrapper around changefeeds. In most cases today, this means opening up a single changefeed and dispatching events to all relevant subscribers. The current Fusion server implementation appears to open a new feed for every subscribe request. As a result, performance is going to degrade (probably rather quickly) as number of client users increases.

Implementation

Since the protocol is pretty simple and the options are fairly limited, it should be quite easy to know when existing open feeds can be used to fulfill new requests. I can only think of a couple things we'd have to to look out for:

1. Can't rely on include_initial optarg. Even if we cached the initial values, they would likely be out of date. I think we'll have to just run a query to get the initial values each time a client subscribes.
2. Logic around when we should close a feed. It's likely that for a given app there will only be a handful of subscriptions. Thus, keeping feeds open for some time after the last client disconnects might make sense as an additional optimization.

pagination api

This is an issue for specifying the pagination api

fusion("table_name").paginate('timestamp', 10).value().then((rows, nextPage) -> {
  // do stuff with rows
  someDomNode.onClick(nextPage().then(...))
})

nextPage returns a promise that's resolved when the next page of results shows up.

This might be funky, an EventEmitter api might be better where you can do .on('page', ...) and you get a function that grabs the next page.

fusion('table_name').page('timestamp', 10).value().then((emitter, nextPage) => {
  emitter.on('page', (rows) => {})
  someDomNode.onClick(() => nextPage())
})

This way the user doesn't have to deal with tokens etc, it's just packed away somewhere

Add support for a mock client-side Fusion server

It is extremely valuable to let people run and tinker with example apps without having to set up any server infrastructure (for example via jsfiddle). Since the Fusion protocol is really simple, we should eventually add a mock client-side Fusion server.

Update examples

I believe currently both examples use the old eventEmitter based subscription model and would not work.

todos.subscribe()
        .on("added", added)
        .on("changed", changed)
        .on("removed", removed)

Ditch EventEmitter

I spoke briefly about this with @coffeemug earlier, and this is not the immediate priority, but I think it would be a good idea.

EventEmitters have a few things that are kind of crappy about them:

1. They make it annoying to unregister listeners. You need to keep the original callback around, there's no other way to look up your listener to remove it. This makes it hard to write abstractions that wrap callbacks from the user since the function the user thinks they registered the event with is different from the function you actually passed to the eventemitter.
2. They allow anyone to emit events on them (.emit is public) and they allow any kind of event to be emitted. This is flexibility we don't need, and it requires being careful about the assumptions we make in the code because a user could do some weird stuff.
3. Delegating events is listener-leak prone. Every subscription emitter needs to be forwarded the "connected" and "disconnected" events from the underlying fusion connection. This means there needs to be a lot of code around to ensure listeners aren't accidentally forgotten about

Proposed replacement

1. The fusion object would have onConnected, onDisconnected and onError methods defined on it. You'd pass them the callback you want, they'd get the fusion object itself in the former two, and the error in onError.
2. .subscribe would return an object with .onAdded, .onRemoved, .onChanged, onSynced onComplete and dispose methods defined on it. Just like the fusion object they receive a callback, and return a function that unregisters the listener.

Examples

let fusion = Fusion(hostString, {secure: false})
let stopListening1 = fusion.onConnected((fusion) => console.log("Woo connected 1"))
let stopListening2 = fusion.onConnected((fusion) => console.log("Woo connected 2"))
stopListening1()
// Sometime later, "Woo connected 2" is logged

Subscriptions:

let subscription = collection.findAll({id: "foo"}).subscribe()
let cleanups = [
    subscription.onAdded((val) => console.log(val)),
    subscription.onRemoved((val) => console.log(val)),
    subscription.onChange((val) => console.log(val.new_val, val.old_val)),
]
// Sometime later
cleanups.each((cleanup) => cleanup())
subscription.dispose()

Creating an rxJs observable (we would likely do this internally in toObservable, but it shows how similar the apis are)

let subscription = collection.findAll({id: "foo"}).subscribe()
let addStream = Rx.Observable.create((observer) => {
    let unregisterAdd = subscription.onAdded(observer.onNext)
    let unregisterError = subscription.onError(observer.onError)
    let unregisterCompleted = subscription.onCompleted(observer.onCompleted)
    // Cleanup function rxJs will call when the observable stream is .dispose 'd
    return () => {
        unregisterAdd()
        unregisterError()
        unregisterCompleted()
        subscription.dispose()
    }
})

Fusion Server can become out of sync with RethinkDB instance if table is manually deleted

At the moment Fusion Server stores metadata about collections in the fusion_internal database under collections. Stopping the server, deleting one of the tables listed there, and then restarting the server does not remove the metadata of that now deleted table from fusion_internal.collections and thus it will not attempt to recreate it.

Not sure about the inverse when deleting the metadata from fusion_internal.collections if it will restore it if the table still exists.

Fuller spec for sync API

In #1 there is a very basic spec for how Fusion will behave. This issue is about designing the full spec for the client-side library for Fusion (just the sync part -- not auth/identity/geo/etc.).

Connecting to the Fusion server:

// Connect to the Fusion server
var ref = new Fusion('http://localhost:8181');

// A user can make multiple references to multiple Fusion servers
var ref2 = new Fusion('http://localhost:9191'); // note: different server
var ref3 = new Fusion('http://localhost:8181'); // note: same server as one above

The rest of the commands below will be accessible from the Fusion object.

Getting a reference to a collection:

Please note: in Fusion we'll be referring to logical data containers as collections, not as tables. In practice a Fusion collection will map directly to a RethinkDB table.

Also note: in Fusion it is not necessary to explicitly create a collection. If the API references a collection that doesn't exist, the Fusion server will automatically create a RethinkDB table under the hood. (This opens up an DOS attack vector, and the security model will need to address this)

// Get a reference to a todo items collection
var items = ref('todo-items');

// Get a reference to a chat messages collection
var messages = ref('chat-messages');

// Get a reference to a channels collection from a different Fusion server
var channels = ref2('channels');

Getting data:

In Fusion everything is a feed by default:

// Subscribe to a feed on a query. Note: `include_initial` is always true,
// this concept isn't exposed in Fusion, and there is no way to turn it off.
var x = items.subscribe(); // returns an EventEmitter
x.on('added', obj=>...)
 .on('removed', obj=>...);

The EventEmitter returned by subscribe will support the following events:

• added: an object has been added to the collection (also included the initial values). The callback receives the added object directly (no new_vals/old_vals).
• removed: an object has been removed from the collection. The callback receives the deleted object directly (no new_vals/old_vals).
• changed: an object has been changed. The callback receives two arguments -- the old document and the new document.
• synced: this event only gets fired once -- after the initial values have been synced. The callback function receives no arguments.

Note -- if the user doesn't subscribe to the changed event, Fusion client will automatically fall back and fire removed and added event to simulate the changed event. This way users can get started by just defining two events, and can add the changed event later.

You can also ask subscribe to only send the initial values, but not the changes:

// Only subscribes to the the initial values. The only available event is `added`,
// and it gets called for each item one by one. This is equivalent to just running
// a RethinkDB query without a feed.
items.subscribe({ updates: false }).on('added', obj=>...);

Finally you can get the full value from the server without subscribing to changes (note: this is an edit):

items.value().then(obj=>...);

Commands to filter and transform data:

We'll support the following commands to filter and transform data:

• find -- the same as ReQL getAll.
• findOne -- the same as ReQL get.
• between -- the same as ReQL between.
• ordered -- equivalent to ReQL orderBy, but no unindexed ordering will be supported.
• limit -- the same as ReQL limit.

Please note: we may or may not rename these commands (given the whole ReQL confusion/compatibility issue).

// Look at a single document
items.findOne(1).subscribe().on('added', obj => ...);

// Look at an ordering of documents by some index
items.ordered('index-name').limit(5).subscribe().on('added', obj => ...);

Commands to modify data:

We'll support the following commands to manipulate data:

• store -- equivalent to .insert(obj, { conflict: 'replace' })('generated_keys').
• update -- equivalent to ReQL update.
• remove -- equivalent to ReQL delete.

These commands will return a promise.

// Inserts the element (and replaces an existing one if it's missing).
// The promise will return a list of ids of all inserted/replaced documents.
items.store(obj).then(ids => ...);

// Same as above, but sets `conflict` to `error` (equivalent to basic `insert`).
items.store(obj, { conflict: 'error' }).then(ids => ...)

// Updates a document and returns an empty promise.
items.findOne(1).update(obj).then(() => ...)

// Remove a single element from the collection.
// Returns an empty promise (hehe).
items.findOne(1).remove().then(()=>...);

Add Observable interface

Following #33, it makes sense to create an Observable interface. Here's how I've currently implemented it, but it's obviously open for changes:

• We detect if the Rx module is available as a global, or if it can be imported through browserify
• If so, we add the following methods to a subscription:
  • observeAdded, observeRemoved, observeChanged, observeConnected, observeDisconnected, observeSynced

Each of these methods can be invoked like:

let sub = query.subscribe()
let removed = sub.observeRemoved()
let added = sub.observeAdded()

The default behavior, when .dispose() is called on any of the observables, is to dispose the underlying subscription. So doing this:

added.dispose()

Will result in the subscription being disposed of. If the intent is not to close the underlying subscription when an observable is disposed of, you can provide a cleanup function. It receives a listener cleanup function that should be called:

let added = sub.observeAdded( detachObservable => {
  // do something before detaching
  detachObservable()
  // do some other cleanup here
})

Babel polyfill

Needs to be added to package.json in client

Stored Javascript Date objects should return as Javascript Date objects.

😢

Automatic data structure sync

@mglukhovsky pointed out that while the Fusion subscription API is really nice, 95% of the effort of using it involves boilerplate code to synchronize the state of the server with a browser-side data structure. He pointed out that the API would be dramatically nicer if there were a higher level primitive to do the synchronization.

We discussed this earlier and decided to put it off, but now that the subscription API has stabilized, I think it's important to ship v1 of Fusion with this synchronization API (since it really does make everything 100x nicer).

The basic idea is to allow attaching subscriptions and observables to objects:

let a = [];
collection.subscribe().attach(a);

// At this point `a` automatically updates to reflect the data on the server

This API is complicated by two issues:

• Users might want to be aware of state with respect to optimistic concurrency (i.e. was the state updated optimistically, or was it actually updated on the server?) It's not clear how to indicate that here.
• In most modern frameworks (like React, Angular, etc.) the user has to notify the framework the state was updated (e.g. via setState in React). This API needs to be designed in a way where following this pattern is doable and convenient.

I'll speak to @deontologician when he gets back before proposing a full API for this (I need to find out the current state of affairs with respect to observables, and how to integrate attaching/detaching nicely with the current API).

Add uglification and other optimizations to gulp client library pipeline

findAll() with a range

I can't figure out how to do a range query, such as between(1, 5,{index: "someIndex"} ) . I asked about it at slack #fusion channel with no luck. @mglukhovsky suggested I write an issue, so I did.

Multiple databases for one Fusion server

The application we're building has one database per tenant. This makes moving our customers from our hosting to self hosting fairly trivial as we only have to move the DB and a copy of the proxy server onto infrastructure that they control. It looks like the database connection in Fusion is a global parameter set per server, not per connection. Is this something that would be able to go into a per connection config (with a default database specified perhaps)?

This also needs to interact with the security API, perhaps part of the connection grant could be checking if the user is authorised for this database.

One alternative for us is to run one Node server per customer. This is possible, but not ideal and will get a bit messy managing host:port config for all of them. It might also be possible to run middleware for each customer, but again that will be a bit messy.

Fusion command line interface

There's two parts for this to think about: the Fusion Server and Fusion Ops. I'm wondering if the functionality should be wrapped up into the same command line interface or it should be kept separate. I think for the simplicity of brew install [fusion|rethinkdb-fuison] it should be kept together.

Like we talked about, it's not going to be too hard to have something better than gcloud or awscli but I think we can take hints from things like rack, tugboat, and maybe Heroku Toolbelt.

What I like best in CLI tools I find is syntaxes like fusion [verb] [object]. Otherwise for simpler actions fusion [object] usually suffices. Unlike some of the aforementioned tools which offer so many options that it's impossible to use but more importantly remember easily. I think some sort of guidelines are needed to make sure we don't enter command argument hell.

This is a very basic list:

fusion server [fusion server flags]

Starts what we have currently as the fusion server. If npm install -g fusion hasn't happened yet, the CLI will do this for you. Also will check and notify the user if their current fusion server is out of date.

fusion deploy [--app-directory directory]

Deploys the current directory (and searches recursively up) until it finds some directory which is somehow designated as a "fusion application".

fusion status app [--app app_name]

Give the status of the application of the CWD or explicitly state the application via command line flag.

fusion status db

Give some sort of status table of the database cluster.

fusion scale app number_servers

Scale up or down the current application to number_servers.

fusion scale db number_dbs

Scale up or down the number of dbs nodes in the Fusion cluster. This is somewhat complicated since this command would potentially affect multiple applications running on the Fusion cluster.

fusion stop app [--app app_name]

fusion start app [--app app_name]

Behavior on socket errors

Currently, if a websocket encounters a (low-level) protocol error, like closing or whatever, I emit a 'disconnected' event on all currently outstanding eventemitters. Is this the right behavior, and should we do something with outstanding promises?

Latency Compensation

Firebase, Relay, and Meteor have built in latency compensation in their client side libraries. Updates are optimistically applied, and then 'committed' client-side once the confirmation comes back from the server.

As we discovered in our RethinkDB websockets project (pre-fusion), the richer your query model, the more difficult latency compensation is. We were running full REQL queries on the client, it is very difficult to simulate updates without something like Reqlite. Firebase has the easiest time of it because their query model is so impoverished.

Related links:

http://info.meteor.com/blog/optimistic-ui-with-meteor-latency-compensation
https://github.com/neumino/reqlite
https://facebook.github.io/relay/docs/guides-mutations.html#optimistic-updates
https://www.firebase.com/docs/web/guide/offline-capabilities.html

Protocol sketch

This is the proto-protocol that we've implemented so far in the client. It's basically a straw-man implementation so feel free to pick it apart. The goals are probably something along these lines:

1. simple
2. easy to read & understand in wire-format
3. query paths easy to parse with .split()

Requests have the following format:

{
  "type": <request type>,
  "requestId": <increasing counter>,
  "data": <subobject depending on type>
}

The types at the moment are

1. STORE_REPLACE
2. STORE_ERROR
3. SUBSCRIBE
4. QUERY
5. UPDATE
6. REMOVE

Request IDs serve the same purpose as tokens in the drivers, correlating requests and responses

QUERY/SUBSCRIBE have a path associated with them. It's similar to the result of .build() on driver terms, except it's not nested and kind of looks like a url. Query terms are separated by a slash and arguments of the term are separated by a colon.

Example:

fusion("items").ordered("age").between(16, 25)
becomes
items/ordered:age/between:16:25:age

fusion("items").between(16,24)
becomes
items/between:16:24:id

A full request for the above:

{
  "type": "SUBSCRIBE",
  "requestId": 2,
  "data": {"path":  "items/between:16:24:id"}
}

The Fusion server should also serve the client library

It'd be wonderful if the Fusion server also served the client library, so it's easy to get started, and the server / client stay in sync, so when you do:

$ fusion --rethinkdb-host localhost --rethinkdb-port port

...a server would start up, and the client library would be available at /fusion/fusion.js.

Socket.IO does this as well, so the client library is available at /socket.io/socket.io.js, and it's quite nice.

server side variables

Had a chat with @danielmewes, he said I should write this. To keep this short, pretty much what the topic says. Would be pretty cool to have server side variables. Obvious example that comes to mind would be what r.now() is in ReQL. Timestamps are tricky because if you don't have "neutral clock", times you use are pretty much what users have set their clocks to. So the precision, even without having someone passing nonsense on purpose, would be pretty low.

Spec out and build an identity management API (e.g. listing/deleting/etc users)

TBD

Getting different results from "onAdded" when using .subscribe()

When using the .subscribe({ onAdded: function(){ ... }, }); syntax I believe I'm seeing all the documents in the table running through this function immediately.

However, when I do the following:

const sub = fusion.subscribe(); 
sub.onAdded = function(added){ ... };

The results are as expected and the function doesn't run until I .store() something. Any thoughts on why this might be?

Here are my full examples:

this.props.fusion.subscribe({
        "onAdded": (function(added){
          console.log("ADDING");console.log(added);

          // Grab current state of messages
          var currentMessages = this.state.messages;

          // Pop off the front to keep us at 10.
          if (currentMessages.length >= 10){
            currentMessages.shift();
          }

          // Set the state with the newest message
          this.setState({
            messages: currentMessages.concat(added)
          });
    }).bind(this)
});

const sub = this.props.fusion.subscribe()
sub.onAdded = function(added){
console.log("ADDING");console.log(added);

// Grab current state of messages
var currentMessages = this.state.messages;

// Pop off the front to keep us at 10.
if (currentMessages.length >= 10){
    currentMessages.shift();
}

// Set the state with the newest message
this.setState({messages: currentMessages.concat(added)});
};

Make sure Fusion server is importable/extensible in Node

@deontologician and @segphault pointed out that once the user outgrows the Fusion API and wants to add custom handlers, the transition to adding a Node.js server that has nothing to do with Fusion is a little jarring. Their suggestion was to make the Fusion server importable in node.js and extensible, which I think is brilliant.

So, basic Fusion server usage would work like this (as before):

$ fusion --rethinkdb-host localhost --rethinkdb-port port

But once the user wants to add custom code, instead of just starting the Fusion server, they'd do it like this. On the serverside:

var fusion = require('fusion');
var app = fusion();

// extend fusion with a custom endpoint
fusion.extend('myCommand', (...) => {
  // custom ReQL/JS code goes here
})

Then in the client/browser:

fusion.myCommand(...).on(...);

This is obviously underspecified; we should think about this and specify it better once everything is further along.

/cc @Tryneus

Avoid self-races in table creation in dev mode

right now multiple tables with the same name can be created simultaneously, and it causes all further queries to fail

HTTP => HTTPS upgrade?

I don't like how a user will have to switch between <script src="https://localhost:8181/fusion.js"></script> and <script src="http://localhost:8181/fusion.js"></script> when switching between --unsecure and not on Fusion server.

Would it be possible to just have http switch to https if --unsecure was not specified to make this simpler? I've done this with Nginx before but am not sure if it's possible from Node-side.