colinskow / pouch-mirror

Helps you maintain a local mirror of any CouchDB database for lightning-fast reads and secure writes.

PouchMirror

Version 0.2.0 has a NEW API, see usage below.

PouchMirror helps you create a local mirror of any CouchDB database for lightning-fast reads and secure writes. It differs from standard CouchDB replication in that the remote DB is the single source of truth. This prevents most write conflicts.

Accessing a remote CouchDB instance can be slow. PouchDB is an incredible tool that allows you to create local instances of your databases in any Javascript environment and keep them in sync with your server. The problem is that writing to a replica of your main database comes with no guarantees and can lead to unexpected results. For example, if multiple nodes are writing to the same documents at the same time, automatic conflict resolution can cause changes you thought you saved to pop out of existence.

When PouchMirror first starts up it will initiate replication to create a local in-memory mirror of your remote database. Before this initial replication is finished, all read requests are automatically routed to the server to ensure accurate reads. Once the initial replication is complete, you will enjoy lightning-fast reads from your local database on all requests.

When you write, PouchMirror makes sure that the data is saved on your CouchDB server with no conflict AND fully replicated to your local instance before resolving the promise. If a conflict arises, your promise will berejected and no data saved.

While PouchMirror does feature browser support, it uses your remote db as the primary source of truth and therefore is not appropriate for front-end apps that need to work offline. If you are building an offline app, NG-Pouch-Mirror is a much better option.

For issues and feature requests visit the issue tracker.

Build status

Usage

PouchMirror is an exact mirror of the PouchDB API, and can serve as a drop-in replacement for your existing PouchDB code. Both promises and callbacks are supported.

In Node.js, simply require "pouch-mirror" and initiate it: new PouchMirror(localDB, remote)

localDB MUST be an instance of PouchDB. remote may be a URL string OR an instance of PouchDB.

In the browser, simply include a script tag below PouchDB: <script src="dist/pouch-mirror.js"></script> and use the same syntax.

Example:

var PouchDB = require('pouchdb');
var memdown = require('memdown');
var PouchMirror = require('pouch-mirror');

var localDB = new PouchDB('testDB', {db: memdown});
var mirror = new PouchMirror(localDB, 'http://localhost:5984/pouchtest');

var replicator = mirror.start({retry: true});
// PouchDB replication events will pass through here.
// When the initial replication is complete you will get a
// special one-time 'up-to-date' event.
replicator.on('up-to-date', function(info) {
  console.log('Congratulations, initial replication of ' +
    info.db + 'complete!');
});

mirror.post({title: "Ziggy Stardust"})
  .then(function(result) {
    return mirror.get(result.id);
  })
  .then(function(doc){
    console.log(doc);
    // You can pause replication any time you want like this:
    mirror.pause();
    // And restart it again with mirror.start()
  })
  .catch(function(err) {
    console.log(err);
  });

API

PouchMirror uses that exact same API as PouchDB, but does some magic in the background to ensure your local mirror stays in perfect sync. In addition, you have the following commands available:

• pouchMirror.start([options]) - starts replication and returns the PouchDB replicator object.
• pouchMirror.pause() - pauses replication. replicator.cancel() also does the same thing.

All start options will be passed directly to PouchDB.replicate:

• options.retry - set this to true if you want PouchMirror to automatically attempt to reconnect in the case of replication problems. This uses PouchDB's default backoff function with a maximum timeout added.
• options.maxTimeout - the retry timeout for the default backoff will never exceed this value (default 600000, 10 min). Set to 0 to allow infinite backoff.
• options.back_off_function - supply your own backoff function. maxTimeout has no effect with this option.

All write requests will ALWAYS go to the remote db first, and the promise will only resolve after changes are confirmed to have replicated to local. When replication is paused, all read requests will also go to remote. After the initial replication has finished all read requests will come from the local db as long as replication stays active.

Road Map

Local first mode

Continuous replication can be request heavy, and in order to save hosting money I have a request to create a mode which puts the local database first and debounces changes before sending them to the remote server.

More extensive testing

I have run the tests against a local CouchDB instance, and have been using it in small-scale production on Cloudant. However, this definitely needs more experience on large setups to make sure it stands up to scaling demands.

pouch-mirror's Issues

Replication blues and the mystery of 'uptodate' status.

Hi Colin,

Aside from Superlogin, I am also trying to integrate pouch-mirror into my application to sync with a logged in user's remote private DB (CouchDB 1.6.1) with an IndexedDB based in-browser PouchDB instance.

My setup is something like this -
Front end - React, Reflux, superlogin-client and pouch-mirror.
Server - Express, Superlogin
DB - Couch 1.6.1 (brand new, Admin Party enabled).

Right now, when I am using pouch-mirror, its able to write docs to the user's remoteDB correctly, when the sync has not finished. However, the sync never finishes. The replication process keeps on throwing 'ETIMEDOUT' exception (caught in the error event handler in startLiveReplication). Due to this issue, the replication to local DB _never _ completes and the application keeps on using remoteDB only.
Using Fiddler, I saw that the request to the Remote URL look like this -

GET http://localhost:5984/<remoteDBname>/_changes?timeout=25000&style=all_docs&feed=longpoll&heartbeat=10000&since=3&limit=100 HTTP/1.1
Host: localhost:5984
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept: application/json
Origin: http://localhost:8080
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Authorization: Basic Wl9zQ0dKZFNSa2FmNm9wOUcxQ0xwZzp0MWlqbUJCQlNJbUMyMVpfV1U1bjln
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8

Its on this request where pouch-mirror receives the time-out exception. I take it that this is supposed to be a long-polling request and wonder whether this exception is expected.

I looked into the code of startLiveReplication and saw that there's a bind to uptodate event which is supposed to handle sync completion as its this place where pouch-mirror switches remoteDB with localDB. However, this enevt never gets invoked. Also, I didn't notice this event in the PouchDB replication docs. There's a complete event listed but per the documentation, for a live replication, the complete event will never be invoked unless the sync is cancelled.

Thoughts!?

PS: My entire development stack is on a single machine.

Add support for `destroy` method.

Currently, the pouch-mirror API doesn't include destroy event. Can I take a PR to add one? The idea is to destroy localDB instance. This is useful in scenarios where multiple users share one browser environment and when a user logs-off, it is prudent to delete any local DB containing private information. As pouch-mirror abstracts the localDB instance, exposing a destroy method will make it easy to delete the localDB.

Endless replication when using reverse-proxy.

Hi Colin,

This is totally a non-issue with library. I needed your help in debugging a problem I am facing and as I didn't have your email, I am reaching out to you here.

To prevent having to expose CouchDB endpoint to my application users, I tried to add a simple reverse-proxy using request in my express app -

```
// Bind /db to couch DB backend
var DATABASE_URL = 'http://localhost:5984';

// middleware itself, preceding any parsers
app.use(function(req, res, next){
    var proxy_path = req.path.match(/^\/db(.*)$/);
    if(proxy_path){
        var db_url = DATABASE_URL + proxy_path[1];
        console.log("db_url: " + db_url);
        req.pipe(request({
        uri: db_url,
        method: req.method
        })).pipe(res);
    } else {
        next();
    }
});

Also, I added dbServer > publicURL: 'http://localhost:8080/db/' in superlogin config, so that the private DB urls come mapped to the `/db` endpoint. Things went good this far. However, when I ran the application, the replication initiated by pouch-mirror went amok and never ended. The calls trough the proxy are going through, however somehow the replication is screwed up. The private DB size ballooned very quickly from mere 8.1k to 1MB in no time.

Do you have experience with a setup like this? Would be great if you can give me any pointers.

Thanks!

Enhancement - Local First Mirrorring

Pouch-mirror uses a remote-first strategy for any edit calls. This ensures that there's only a single source of truth in a multi-client-single-server scenario and also helps with conflict resolution.
But in cases where the client interactions are heavy, the server-first approach may lead to a slower turn-around, and may not work well in cases of flaky network connectivity.

Would you plan to revisit this project to bake-in a local-first strategy? If you are too busy now, I can fork this project and add one.

For a local-first approach, I think the following interactions should be acceptable -

1. On application setup, do a one-time two-way sync with Remote. During this time, redirect all read/writes to the server.
2. Once the sync is complete, switch to local DB. Now all r/w operations happen locally.
3. On every write (to local), schedule a (delayed, say 2-4 seconds) one-time 2-way sync with remote. On conflicts, raise events.
4. If another write occurs before the delayed sync happens, cancel the sync and reschedule it. This way if a lot of writes occur in succession, we don't end-up doing several sync's with the server. Basically we wait for things to cool down on the client before firing up the remote sync.
5. Before local destroy, offer a configurable replication to remote. This way scenarios like 'logout' which invoke destroy can rest in peace knowing the localDB will replicate to remote before being wiped.

Thoughts!?