Fuller spec for sync API about horizon

[marshall007]

Thoughts on .toArray():

I can see value in having a squash-like option to subscribe, but at first glance .subscribe({ updates: false }).toArray() seems a little odd. Something like this might be a bit simpler:

collection.toArray() // -> Promise
collection.subscribe() // -> EventEmitter
collection.subscribe().toArray() // -> Error / not a function

Consider also subscriptions to single-document changes. In this case, toArray doesn't make a lot of sense. Maybe we should call it toPromise so we can have the same method name to get initial values in either case?

Instance methods:

I don't wanna get too crazy here with the first pass, but it would also be cool if the objects resolved from toArray/toPromise were wrapped in the Fusion API so you get some handy instance methods:

collection.findOne(1).toPromise().then(item => {
  item.foo = 'bar';
  item.update(); // -> Promise

  item.subscribe(); // -> EventEmitter
})

[coffeemug]

at first glance .subscribe({ updates: false }).toArray() seems a little odd

The rationale for it was to make it easier to build realtime apps than non-realtime ones by default, so we thought we'd make it a little harder to just get the value by going through the subscribe interface. I agree that the name toArray doesn't make sense -- we should call the method toValue instead.

So users could do:

items.subscribe({ updates: false }).toValue().then(obj=>...);

You're right though that we might be trying to be too clever for our own good, and maybe the proper interface for it is just this:

items.value().then(obj=>...);

That's probably better. What do you think about the argument that getting the value should be a little hard to encourage people to fall into realtime apps by default?

it would also be cool if the objects resolved from toArray/toPromise were wrapped in the Fusion API so you get some handy instance methods

I agree that's cool. I'd leave it for v2 though -- I think it's overkill for the first version.

[marshall007]

👍 to toValue, I like that better than toPromise.

What do you think about the argument that getting the value should be a little hard to encourage people to fall into realtime apps by default?

I see what you're saying. I think the main reason why we (speaking generally as developers) don't make our apps realtime is because the tools for doing so either don't integrate well with our existing backends or aren't mature enough to make us feel comfortable adopting them. If Fusion provides solid, easy-to-use APIs for build realtime apps, people will naturally use them. I don't think making the developer type a few more characters would be a strong influence in that decision.

In fact, if I'm considering migrating my existing non-realtime apps I'd prefer a solution that makes that transition easy; taking into account the intermediary stage where I can start by first migrating to Fusion then iteratively introduce realtime behavior as it makes sense.

I think Firebase is a good illustration of this problem. It's super great if you're building an app from scratch, but really sucks if you're trying to transition.

[coffeemug]

👍

[mlucy]

Also note: in Fusion it is not necessary to explicitly create a collection. If the API references a collection that doesn't exist, the Fusion server will automatically create a RethinkDB table under the hood.

We're currently planning to have a development mode with no security restrictions, and a production mode with security restrictions turned on, right? I think allowing automatic table creation in debug mode is probably fine. Allowing automatic table creation in production mode seems like a bad idea unless we change the server to make table creation faster and conflict-proof.

// Subscribe to a feed on a query. Note: include_initial is always true,
// this concept isn't exposed in Fusion, and there is no way to turn it off.

That seems like a bad idea to me. There are lots of cases where you'd only want to get changes and not to get initial values. (E.g. a chat app where you don't want to load the whole history, you only want to see new messages, or a monitoring app where you just want to see events as they happen and don't care about the whole history of events.) It's also essentially 0 development effort to just accept an argument we pass through to the server. (I think making include_initial be the default is a good idea, though.)

The EventEmitter returned by subscribe will support the following events:

We should probably also have events for when we start receiving changes and when we stop receiving changes (either because we disconnected or because the feed was closed). Maybe the latter should be an error handler.

(The reason to have an event for when you start receiving changes is that it can take arbitrarily long for the first change to come in, and you want to be able to stop your loading animation or whatever as soon as things are actually working.)

subscribe({updates: false})/value

It seems like subscribe({updates: false}) and value are solving the same problem and that value is solving it better. Why do we have both?

(Also, as an aside, subscribe({updates: false}) will still have to handle uninitial vals unless we try to do something really clever, so you'll still have to register an on('removed') handler.)

ordered

Could we maybe call this order so it matches the tense of the other commands? (E.g. it's not found or limited.)

Also, how do we do ascending/descending ordering? This would be a great time to murder r.asc and r.desc.

update

How would you atomically increment a counter with this API? I think that class of task is common enough that it needs to be easy even in the first version.

Also, we should probably have an equivalent of return_changes in this API since set-and-get is such a common thing to want to do.

---

A few other thoughts:

• How does error handling work? If a write or read fails, how is that reported? How are disconnects reported?
• Is squash on by default or off by default? Is there a way to toggle it?
• Firebase supports generic atomic updates, are we going to try and support that?

[coffeemug]

We're currently planning to have a development mode with no security restrictions, and a production mode with security restrictions turned on, right?

Yes, that's the idea.

There are lots of cases where you'd only want to get changes and not to get initial values.

Ok, that's a pretty good point. We can add that as an option.

We should probably also have events for when we start receiving changes and when we stop receiving changes

See #9.

It seems like subscribe({updates: false}) and value are solving the same problem and that value is solving it better. Why do we have both? (Also, as an aside, subscribe({updates: false}) will still have to handle uninitial vals unless we try to do something really clever, so you'll still have to register an on('removed') handler.)

Think of subscribe({updates: false}) as exactly the same thing as value(), except that goes through the EventEmitter API and not a json document/array. The reason to do it is that users can write code to process events/insert things into the DOM in exactly the same way regardless of whether they want updates or not.

Also note that with subscribe({updates: false}) people won't need to deal with uninitial vals, because the Fusion server will just run it as a RethinkDB query, not as a changefeed.

[coffeemug]

How does error handling work? If a write or read fails, how is that reported? How are disconnects reported?

For disconnects, see #9. For writes, JS promises have provisions for it (e.g. you say .then().error()). This is a bit underspecified in the above spec (it's not clear exactly what object the user will receive on error) -- we'll have to figure this bit out.

Is squash on by default or off by default? Is there a way to toggle it?

I think it should be on by default, and there shouldn't be a way to toggle it.

Firebase supports generic atomic updates, are we going to try and support that?

I'd like to open a separate issue for this once we get a basic implementation up and running, since this is a sufficiently tricky matter. But yes, I think we'll need to do something about it in v1.

[mlucy]

I think it should be on by default, and there shouldn't be a way to toggle it.

In general I think we should allow some way of passing through all the options changefeeds currently take. The extra development effort is negligible, and it might turn out someone needs it.

In this particular case, I think monitoring apps could very well want to turn squashing off so they see every event.

[deontologician]

Also, how do we do ascending/descending ordering? This would be a great time to murder r.asc and r.desc.

Currently, you just do .ordered(fieldName, direction: 'desc')

[coffeemug]

Currently, you just do .ordered(fieldName, direction: 'desc')

👍 for this API (though we might want to turn the flag into a boolean)

[deontologician]

Yeah, that sounds better, I'll do that

[coffeemug]

In general I think we should allow some way of passing through all the options changefeeds currently take.

I don't have a strong opinion on this yet, but my current (weakly held) hypothesis is that we shouldn't do that unless there is a very good reason. The issue here is restricting the API in a such a way that people can get started very easily, and smoothly start adding ReQL as their requirements get complicated. I think complicating the API is dangerous for Fusion, and we should be careful about doing too much of it until we understand how people want to use it.

[marshall007]

Are you guys planning on supporting joins in the first implementation?

Also, I totally get why transformation (and most) aggregation functions definitely shouldn't be part of the Fusion API, but I think count would be an exception. Obviously this isn't even supported in ReQL changefeeds yet, but I found lack of count support to be a huge limitation when working with Firebase. Made it almost impossible to build useful dashboards.

[coffeemug]

Yeah, we were talking about joins and counts -- that's still up for grabs. I'll open a separate issue for those (as well as atomic updates).

[deontologician]

This issue is a bit out of date and doesn't really have a criteria for being completed, so I'm closing it. The protocol document now lives at https://github.com/rethinkdb/fusion/blob/master/docs/protocol.md