pagination api about horizon

[marshall007]

I did some fancy things when implementing client-side paging services over our REST APIs, some of which might be useful here:

// `paged` is an array-like object which gets updated in-place.
// This works really well with two-way data binding and I imagine
// would be especially awesome with changefeeds.

var paged = fusion('table_name').paginate('timestamp', 10).value();

// The following operations update `paged` in-place once the request is resolved.
// This is great for the "fire and forget" use-case of binding click events.

paged.next(); // -> `paged`
paged.prev(); // -> `paged`

// When you need to do more complicated stuff, like loading indicators,
// the promise is still accessible.

paged.$future; // a `Promise` of the last async operation
paged.next().$future.then(() => {
  // At this point we know, `paged` has been updated.
})

[marshall007]

It would also be useful to have access to the total number of rows and/or pages. If this is included, we'd also have to think about how the client gets notified when those values change on the server.

Additionally, we will need to consider how this API plays with changes in general. What happens when rows get added/removed from the currently visible page boundary? This all seems intimately related to #52, but with addition things to consider since the "subscription" object would need to inherit the pagination API methods.

[deontologician]

total rows would require a .count() which isn't really performant. I would definitely make that optional if possible.

The paging api was going to need to do a between with limit on the first query. And it needs to create an s-index on both the page field and the id, so that the next query can start with the last row received + 1. This still leaves changefeed updates in a weird spot. A lot of pagination apis ignore incongruities between pages, but it would be nice if we had some way of dealing with it if it isn't too hard or too slow

[marshall007]

total rows would require a .count() which isn't really performant. I would definitely make that optional if possible.

Agreed, it actually makes more sense to have the user make that call separately if they need it. That would just be part of the count and joins thing discussed in #8 (comment). I don't think a separate issue was ever created for it though.

---

A lot of pagination apis ignore incongruities between pages

@deontologician are you talking about ignoring changes related to row ordering/page boundaries, other updates to unrelated fields, or both? I think there are different use-cases where each behavior would be desirable:

1. Basic paged table where application doesn't care about any realtime features (neither)
2. Table where the user is scanning/paging through the UI manually. Changes to which rows are currently visible might be annoying (doc updates only)
3. Event/metrics-related dashboards, collaborative spreadsheet editors, etc (doc updates + ordering)

The .value() and .subscribe() conventions should work well here too for (1) and (2/3) respectively. At minimum, (2) is going to be really important if you want any realtime features on reasonably sized collections. I think (3) could be punted on for v1, but it's still going to be pretty important for most dashboard-related use cases.

Finally, in situations like (3) I can see it being really useful if you could easily toggle back and forth between the behavior of (2) and (3). That way it's easy to "lock in" your current range of documents but continue to see changes to individual rows (like when the user starts editing a row).

---

I really think #52 is going to be a huge piece in tying everything together and making the realtime APIs easy to use on the client. Managing the state yourself, particularly when sorting and paging are involved, is probably the most annoying thing to deal with in realtime apps. If you guys nail that it will be incredible.

[deontologician]

I was talking about the fact that generally paginated requests are stateless, so in between when you send a request for the first page, and a request for the next one, some items may have moved around or will be skipped at the boundary. If you have realtime updates to your current page then I think this problem is solved. Actually, if this works well I think it'd be a killer feature.

If I have this straight:

1. would be just paging with normal queries, no changefeeds (.value)
2. would be paging with initial vals and changefeeds updating what's visible (.subscribe)

What's 3 then?

[marshall007]

@deontologician you're right and I just got a little confused by what you meant. What I was talking about with (1) and (3) can pretty much already be accomplished as you described (with .value and .subscribe respectively).

With (2) the idea was that you could ignore changes that add/remove rows from the page boundary, but continue to get changes that update the currently visible rows. So, after thinking about it, this is the one that might be more difficult to implement. On the server, I think you'd do the same .between(...).limit() to get initial vals, but then a .getAll(...).changes() for updates. This is basically a special case of (3) whereby the paging itself is "stateless", but you continue to receive updates to the rows that were within bounds at the time the query was run.

[marshall007]

Note if rethinkdb/rethinkdb#4909 were implemented, we could just use .orderBy(...).slice(n, m) and forget about all the logic for building a .between(...) query. All we would need to keep track of and update is the numeric starting index. It would also allow us to more easily expose a method for jumping to a particular page in addition to next/prev.

[lcir]

What is the status of this issue?

[deontologician]

This is still in medium-term plans. We want to do this, but a lot of core things need to be implemented first

[almstrand]

If rethinkdb/rethinkdb#4909 is implemented, can the existing Collection API be modified to support an offset function? For example:

users.order("postCount", "descending").offset(5).limit(10).fetch();

or perhaps

users.order("postCount", "descending").slice(5, 10).fetch();

[sunjay]

@almstrand Are you looking for skip()?

https://www.rethinkdb.com/api/javascript/#skip

Edit: This is referring to horizon, sorry. That's a rethinkdb function.