nice-registry / all-the-packages

:package: All the npm registry metadata as an offline event stream. [DEPRECATED]

all-the-packages

All the npm registry metadata as an offline event stream.

Why?

See https://github.com/nice-registry/about#why

Installation

npm install all-the-packages --save

When you install this package, a postinstall script downloads the npm registry metadata to a local JSON file, which is about 540 MB.

To refresh the module once you've already installed it, just reinstall:

npm i all-the-packages@latest

Usage

This package provides a simple event emitter that emits two events: package and end.

const registry = require('all-the-packages')

registry
  .on('package', function (pkg) {
    console.log(`${pkg.name} - ${pkg.description}\n`)
  })
  .on('end', function () {
    // done
  })

To get cleaner package data, use nice-package:

const registry = require('all-the-packages')
const NicePackage = require('nice-package')
const coolPackages = []

registry
  .on('package', function (pkg) {
    if (pkg.name.match('cool') || pkg.description.match('cool')) {
      coolPackages.push(new NicePackage(pkg))
    }
  })
  .on('end', function () {
    // done
  })

Why Streams?

Streams are cool and all, but the main reason this module has a streaming interface is RAM. It would be nice to just require the module and have a giant registry object to work with, but a 400 MB JSON file is too big to fit in memory on most computers.

See Also

• package-stream: An endless stream of clean package data from the npm registry.
• nice-package: Clean up messy package metadata from the npm registry

Tests

npm install
npm test

Dependencies

• JSONStream: rawStream.pipe(JSONStream.parse()).pipe(streamOfObjects)
• event-stream: construct pipes of streams of events

Dev Dependencies

• tap-spec: Formatted TAP output like Mocha's spec reporter
• tape: tap-producing test harness for node and browsers

License

MIT

Generated by package-json-to-readme

all-the-packages's Issues

A few invalid "test" packages in the registry

Not sure if valid bug, or if it's your job to be the package sheriff, but I got a couple errors due to my garbage coding, because it turns out now all packages in the npm registry have a name (go figure).

Steps to reproduce:

const registry = require('all-the-packages');

filterByName(/^eslint-plugin-/i)
  .then((packages) => console.log(packages.length))
  .catch(err => console.error(err));

function filterByName(name) {
  return new Promise((resolve) => {
    const packages = [];
    registry.on('package', (pkg) => {
      if (pkg.name && pkg.name.match(name)) {
        packages.push(pkg);
      }
      if (!pkg.name) {
        console.log("DEBUG:", pkg);
      }
    }).on('end', () => resolve(packages));
  });
}

Output:

$ time node index

DEBUG: { dist: { tarball: 'http://registry.npmjs.org/testx2/-/a' },
  maintainers: [ { name: 'testx5', email: '[email protected]' } ],
  _npmUser: { name: 'testx5', email: '[email protected]' } }

DEBUG: { dist: { tarball: 'http://registry.npmjs.org/zachtestproject2/-/test.tgz' },
  maintainers: [ { name: 'zlrenner', email: '[email protected]' } ],
  _npmUser: { name: 'zlrenner', email: '[email protected]' } }

DEBUG: { dist:
   { tarball: 'http://registry.npmjs.org/zachtestproject3/-/test.tgz',
     _from: 'http://zachrenner.com/test.tgz' },
  maintainers: [ { name: 'zlrenner', email: '[email protected]' } ],
  _npmUser: { name: 'zlrenner', email: '[email protected]' } }

DEBUG: { dist:
   { tarball: 'http://registry.npmjs.org/zachtestproject4/-/test.tgz',
     _from: 'http://zachrenner.com/test.tgz',
     _shasum: 'asd' },
  maintainers: [ { name: 'zlrenner', email: '[email protected]' } ],
  _npmUser: { name: 'zlrenner', email: '[email protected]' } }

346

# node index  51.87s user 1.22s system 100% cpu 52.707 total

Anyways, not sure if you want to emit the package event only if the pkg.value in question passes some basic schema (ie: has a name and version property). 🤷

Where are all the READMEs?

I'm running this piece of code to try and count what percent of packages in the registry have README files.

let haves = 0
let total = 0

registry.on('package', (pkg) => {
  total++
  if (pkg.readme) haves++
  if (total % 1000 === 0) console.log(`${haves} / ${total}    ${(100 * haves / total).toFixed(1)}%`)
})
registry.on('end', () => {
  console.log(`${haves} / ${total}`)
  console.log('percent =', (100 * haves / total).toFixed(2))
  process.stderr.write('DONE\n')
  process.exit(0)
})

Now if I use

const registry = require('all-the-packages')

I get one answer: 25394 / 412075 or about 6.2%

But if I use

const registry = require('package-stream')()

I get a completely different answer. Right now I'm only about 1/4 of the way through (100,000 packages) but it's at 95666 READMEs or 95%.

What's going on?

Stop using skimdb

http://blog.npmjs.org/post/157615772423/deprecating-the-all-registry-endpoint

In six months, on September 1 2017, we plan to shut down the servers that back the GET /-/all endpoint of registry.npmjs.org.

This package will need to be updated to use a different technique for collecting package data at install time, probably using package-stream.

Thanks to @rmg for pointing this out in https://github.com/npm/registry/issues/136