pouchdb / npm-browser

npm in your browser. Because why not. (UNMAINTAINED)

NPM Browser

NPM Browser is an offline-first HTML5 app that replicates most of NPM into your browser. It's designed as a demo showcase for PouchDB.

Is this for real?

Yes, if you wait long enough, it will actually store all of NPM's metadata locally. You can then search it offline.

We're not suggesting that storing nearly 1GB in your user's browser is a good idea. The point is to demonstrate that browsers are capable enough today to handle that load. In a normal PouchDB application, we expect you'll have much less data.

Why does Safari ask for 2GB up front?

If you want to store more than 5MB data in WebSQL, Safari requires that you ask permission from the user. The other browsers are more lenient.

Details can be found in the PouchDB FAQs and "common errors" page.

How it works

NPM is largely built on top of CouchDB. In fact, the main repository data is stored in two databases: the skim DB and the fullfat DB:

• https://skimdb.npmjs.com/registry (skim)
• https://registry.npmjs.com/ (fullfat)

The skim DB contains only project metadata (e.g. versions, maintainers, READMEs), whereas the fullfat DB also contains tarball binaries.

In any case, since CouchDB uses the same replication protocol as PouchDB, the NPM Browser replicates the skim DB to your browser.

PouchDB tricks

Since even the skim DB is a huge database (as of this writing, the .couch file is ~600MB), we needed some tricks in order to get the app to perform speedily and efficiently.

filter-pouch

filter-pouch is used to filter incoming documents and trim away unneeded data before storing it in the local database. This ensures that we don't quickly reach browser usage limits, and that in browsers where we have to request data upfront (i.e. Safari), we don't have to request very much.

pouchdb-load

pouchdb-load and pouchdb-dump-cli were used so that the initial replication doesn't take an inordinate amount of time.

CouchDB replication is pretty chatty, so we work around this by decomposing it into a replication stream which we can then store as plaintext static files hosted on Amazon S3.

Once initial replication is complete, the app switches over to regular replication. As modules are added, modified, and deleted, the app should update in realtime.

Skim DB mirror

Unfortunately, since SkimDB still doesn't support CORS (as of this writing), we have a simple IrisCouch mirror set up at http://skimdb.iriscouch.com/registry. Using the real Skim DB is a TODO.

WebSQL

This app prefers WebSQL to IndexedDB because in Chrome it appears to perform better.

AppCache tricks

We use AppCache so that the app can work offline. Add it as a bookmark to your home screen, and notice how it still continues to work! There are two parts to this:

Updating AppCache

AppCache is, to put it mildly, an unsavory fellow at times. To work with it, we just update a comment in the appcache file whenever we make a change, to ensure that the browser downloads the latest version.

Native homescreen icons

In order to get the right icon added to the homescreen on iOS, Android, and FirefoxOS, icons are specified in the index.html using the Apple format, as well as in a manifest.webapp file, using the FirefoxOS format. (Android will just use the Apple icons.)

npm-browser's Issues

app cache

The "turn off your browser" feature isn't complete until we get rid of the little dinosaur.

Firefox OS - "Access to restricted URI denied"

OK, so CORS is correctly enabled on both Amazon S3 and IrisCouch, but when I install as a packaged app on Firefox OS 2.2, I get:

DOMException [NS_ERROR_DOM_BAD_URI: "Access to restricted URI denied"
code: 1012
nsresult: 0x805303f4
location: app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:234]

ajax@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:234:4
[5]</exports.load<@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:511:0
[7]</exports.toPromise/</promise<@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:763:8
[7]</exports.toPromise/<@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:751:18
[7]</exports.getArguments/<@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/deps/pouchdb.load.js:731:11
doInitialReplication/</</series<@app://2df479aa-8302-3244-a353-3bcc18a595e7/scripts/services/pouch-service.js:97:17

I shouldn't have to use systemXHR because CORS is configured on the server, right?

error during replication: CORS request blocked.

Came across npm-browser as a sweet demo of pouchdb, but when I opened it in Firefox (33) nothing happened. Opening the console gives me this (repeating forever):

Chrome works fine.

Add a full-text search field

I basically wrote pouchdb-quick-search for this app, should integrate it.

Can't fully replicate in Chrome

With both WebSQL and IndexedDB, Chrome stops replicating after about 95% with no error message. Seems there's a hard storage limit in Chrome, since it works in Firefox and Safari.

I'm thinking we need a Pouch plugin called FilterPouch that applies a transform to all incoming and outgoing documents. We can then implement compression on top of that, or maybe just omit the fields we don't need. (NPM has a shit-ton of metadata that is not strictly required for a web UI.)

Complaints about private info showing up in search results

I keep getting complaints that the npm dumps hosted on s3 contain email addresses and such. Clearly someone published those to npm, and I tried to add /robots.txt to ignore them, but it seems to not be working. Maybe it's not worth keeping this webapp live anymore.

Sequence Mismatch

Hi, I tried out the npm-browser.com site today and I ran into an error that prevents the replication from finishing in my case.

Here is what it is trying to do:
https://nolan.cloudant.com/skimdb-mirror/_changes?timeout=25000&style=all_docs&heartbeat=10000&since=161869&limit=500

And the response:
{"error":"bad_request","reason":"Malformed sequence supplied in 'since' parameter."}

The problem appears to be that there is a mismatch between the sequence numbers that my browser has so far and the sequence numbers that the cloudant skimdb-mirror has.

The cloudant sequence numbers are super long and ugly. Here is a working example:
https://nolan.cloudant.com/skimdb-mirror/_changes?timeout=25000&style=all_docs&heartbeat=10000&since=506-g1AAAAG3eJzLYWBgYMlgTmFQSklKzi9KdUhJMjTSy83PzyvOyMxL1UvOyS9NScwr0ctLLckBqmVKZEiy____f1YGc2JNLlCA3cI42cTE2JAoE9AsMsBnUZIDkEyqh9pVAbYrLdnQLC3NhChDQHYpwpWZ47MqjwVIMjQAKaBt-xFeMzQ2SEoySyHGHFTbjAnbdgBiG9hzXWDbTEyNE83NjIkxJwsAmUyRgQ&limit=500

FirefoxDeveloperEdition 35: progress blocked at 19%

In FirefoxDeveloperEdition 35.0a2 (2014-11-17), my progress stops at about 19% and it can't proceed any further. Seems like Firefox no longer shows the "would you like to allow more data" popup?

The error is:

DOMException [InvalidStateError: "A mutation operation was attempted on a database that did not allow mutations."
code: 11
nsresult: 0x80660006
location: http://www.npm-browser.com/scripts/deps/pouchdb.js:3132]

Better error when you're out of storage space

On Chrome for Android it looks like this:

{"message":"there was not enough remaining storage space, or the storage quota was reached and the user declined to allow more space","name":"SQLError"}

Currently the app just says "you've gone offline," which is not helpful.

couchapp

if we change the url line to be

 $scope.couchdbUrl = location.protocol+ '//' +location.host + '/' + location.pathname.split('/')[1];

then we can just stick it in a couchapp and not worry about origin