hpcloud / npm-lazy-mirror

A lazy mirroring server for npm

#npm-lazy-mirror

A lazy mirroring local npm server.

About

This package provides a lazy mirroring option for those that:

• Don't want to mirror the entire couchDB for npmjs.org
• Don't want to setup cron jobs for those tasks
• Want to use an in-memory caching server (allocation configurable)
• Want to easily cache dist tarballs and package metadata to disk

Install

• npm install -g npm-lazy-mirror

Run

With CLI flags:

• npm-lazy-mirror -p <port> -a <remote_address> -b <bind_address> --cache_dir /npm-data

With a JSON configuration:

• npm-lazy-mirror -C /path/to/config,json

See example/server-config.json for usage.

Note: Your remote_address configuration is important, as it is the address used when re-writing tarball URLs in the metadata. It's certainly always best to use a DNS entry here, rather than an IP.

Client configuration

Simply point your local npm config to the lazy mirror (permanent):

npm config set registry http://localhost:2000/

or per install:

npm i --registry http://localhost:2000 supertest

Help

Run npm-lazy-mirror -h to see a full list of options.

Features

• Caches all tarball / JSON metadata to disk
• Mirror serves files (200MB max by default) from memory, with a configurable LRU cache.
• Ability to blacklist packages by semantic versioning specification
• Option to serve stale resources while the upstream registry is offline
• Upstream resources are fetched on the fly from the remote registry, the fetching, storing and serving to the client all happen in the same request.
• Configurable with custom npm registries.
• HTTP/S proxy support
• It's Fast and stands up under load. Expect 5000+ req/s with one core.

A cold run installing express takes ~12 seconds (fetching from upstream registry on-the-fly):

npm install express  2.44s user 0.81s system 27% cpu 11.769 total

A warm run after all express assets are locally cached takes ~3 seconds:

npm install express  2.43s user 0.78s system 115% cpu 2.768 total

Implementation Caveats

You cannot use this mirror for publishing modules or user management, such requests will be forwarded to the upstream registry for processing.

npm-lazy-mirror's Issues

npm-lazy-mirror fails on fetching expect.js

When installing expect.js via np-mlazy-mirror the client (wehre we want to install expect.js) tries to fetch from registry.npmjs.org instead of npm-lazy-mirror.
Client can not access internet directly.
HTTP Proxy forward is done within npm-lazy-mirror.

On CLI:

npm install --registry=http://npm-1.<domain>:8080/ expect.js
npm http GET http://npm-1.<domain>:8080/expect.js
npm http 304 http://npm-1.<domain>:8080/expect.js
npm http GET http://registry.npmjs.org/expect.js/-/expect.js-0.2.0.tgz
npm ERR! fetch failed http://registry.npmjs.org/expect.js/-/expect.js-0.2.0.tgz

Log from npm-lazy-mirror:

[Fri Jan 24 2014 13:04:03 GMT+0100 (CET)] INFO GET /-/all/since?stale=update_after&startkey=1390565030225
[Fri Jan 24 2014 13:04:03 GMT+0100 (CET)] INFO proxying request to registry GET /-/all/since?stale=update_after&startkey=1390565030225 registry
.npmjs.org
[Fri Jan 24 2014 13:04:30 GMT+0100 (CET)] INFO GET /expect.js
[Fri Jan 24 2014 13:04:30 GMT+0100 (CET)] INFO proxying request to registry GET /expect.js registry.npmjs.org
[Fri Jan 24 2014 13:05:55 GMT+0100 (CET)] INFO GET /expect.js
[Fri Jan 24 2014 13:05:55 GMT+0100 (CET)] INFO proxying request to registry GET /expect.js registry.npmjs.org
[Fri Jan 24 2014 13:06:54 GMT+0100 (CET)] INFO GET /expect.js

npm-lazy-mirror config:

NPM_LAZY_MIRROR_PORT="8080"
NPM_LAZY_MIRROR_REMOTE="npm-1.<domain>"
NPM_LAZY_MIRROR_IP="10.175.24.53"
NPM_LAZY_MIRROR_HTTP_PROXY="http://proxy.<domain>:8080"
NPM_LAZY_MIRROR_HTTPS_PROXY="http://proxy.<domain>:8080"
NPM_LAZY_MIRROR_MEM_CACHE_SIZE="1200"

Fail to strip upstream_db_path from package urls

Hello,

The NPM mirror that I'm using has a /npm at the end of the url, so I used upstream_db_path in the config.json to make it work. However, when npm-lazy-mirror proxies back the URLs for the packages, it doesn't strip the /npm out, causing them to 404.

I devised a temporary workaround using nginx (since I'm reverse proxying through that anyways) as follows:

server {
  listen 8080;
  location / {
    proxy_pass http://localhost:20000;
  }
  location ~* /npm(/.*) {
    return 301 $1;
  }
}

The proxy should strip the upstream_db_path from the path to the packages before pushing to the client.

terminating parent process does not kill child

I' trying to run npm-lazy-miror via supervisord.
Supervisord sends a signal to the PID it created.
When starting npm-lazy-mirror forkes another process.
Unfortunately this forked process does not get killed when the master process dies.
This results in supervisord being unable to stop/restart npm-lazy-mirror.

Publishing modules not supported by design?

In the very end, the documentation tells you under Caveats:

You cannot use this mirror for publishing modules or user management, such requests will be forwarded to the upstream registry for processing.

Is this a decision by design, or is it by accident? In other words: Shall the current behavior stay this way (which I actually hope for), or do you plan to change it?

Cannot install [email protected]

This is a follow up to #7
With #7 we can
npm install dateformat
and get the correct [email protected]
but when installing an older version (as karma does), it will not work.
try with
npm install [email protected]

Cannot install wordwrap

I think the problem is, that wordwrap has no release on github.
(Btw. is a strange module. Absolute small and 40.000 downloads per day)

npm ERR! Error: failed to fetch from registry: wordwrap
npm ERR! at RegClient. (/usr/lib/node_modules/npm-registry-client/lib/get.js:134:18)
npm ERR! at cb (/usr/lib/node_modules/npm-registry-client/lib/request.js:27:9)
npm ERR! at RegClient. (/usr/lib/node_modules/npm-registry-client/lib/request.js:122:16)
npm ERR! at cb (/usr/lib/node_modules/npm-registry-client/lib/request.js:158:9)
npm ERR! at RegClient. (/usr/lib/node_modules/npm-registry-client/lib/request.js:303:12)
npm ERR! at Request.self.callback (/usr/lib/node_modules/request/index.js:148:22)
npm ERR! at Request.EventEmitter.emit (events.js:98:17)
npm ERR! at Request. (/usr/lib/node_modules/request/index.js:891:14)
npm ERR! at Request.EventEmitter.emit (events.js:117:20)
npm ERR! at IncomingMessage. (/usr/lib/node_modules/request/index.js:842:12)
npm ERR! If you need help, you may report this log at:
npm ERR! http://github.com/isaacs/npm/issues
npm ERR! or email it to:
npm ERR! [email protected]

Randomly consume all CPU ressources

My npm-lazy-mirror after few days is taking 100% of the CPU slot allowed.

When I restart it, everything became normal, and cpu use is not more few % (even under sollicitation)

Is anyone experiment the same issue?

I continue investigation

Impossible to disable http server

I'm in trouble to desactivate http server

$ npm-lazy-mirror --http_enable false
[Mon Dec 16 2013 15:37:06 GMT+0000 (UTC)] INFO Lazy mirror (HTTP) is listening @ 127.0.0.1:2000 External host: localhost
[Mon Dec 16 2013 15:37:06 GMT+0000 (UTC)] INFO Lazy mirror (HTTPS) is listening @ 127.0.0.1:2001 External host: localhost

The code seems to overwrite arguments or config file. (argv || config || true)

Trying few solutions

Looking

npm-lazy-mirror fails on fetching packages with multiple version in package name

[Thu Jan 30 2014 10:12:13 GMT+0100 (CET)] INFO GET /dateformat/-/dateformat-1.0.7-1.2.3.tgz
[Thu Jan 30 2014 10:12:20 GMT+0100 (CET)] INFO Resource not found: /dateformat/-/dateformat-1.0.7-1.2.3.tgz
[Thu Jan 30 2014 10:13:24 GMT+0100 (CET)] INFO GET /dateformat

`npm-lazy-mirror` global command broken on Windows

$ npm-lazy-mirror

module.js:340
    throw err;
          ^
Error: Cannot find module 'c:\Users\rmcguir\AppData\Roaming\npm\node_modules\npm-lazy-mirror\lib\node_modules\npm-lazy-mirror\server.js'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

Meaning of command-line switches?

When configuring npm-lazy-mirror via CLI switches, the documentation tells you to call it like this:

npm-lazy-mirror -p <port> -a <remote_address> -b <bind_address> --cache_dir /npm-data

Now, what do the various arguments mean? -p and --cache_dir are pretty obvious, I think but what do -a and -b mean, and what are they good for? Are they mandatory? And either way, what addresses do I have to provide here?

It would be nice if the documentation could explain this with a little more detail.

Use Https to connect registry.npmjs.org

Hello.

A issue is the connection between nodejs and npm-lazy-mirror.
An attacker could intercept and rcorrupt a packet before delivering in the cache

@jdpaton it's an issue for me, What's you opinion?
More generally, what's you futur expectation for this soft?

Should we keep an http acces?

For secure it, we must :
Use https instead http, Some code modification here because http was hard coded.
Verify the certificate selfsigned use : Probably by providing it manually.

Maybe I'll try to fix it in next 2 weeks, if anyone want before, do it, and thank's.

Lucas

No cache for scoped packages

The mirror does not retain packages using the scoped package syntax, such as @babel/cli.

Add support for remote proxy usage

I planned to use npm-lazy-mirror in an internal network where all systems have to use a proxy for outbound connections.
The npm-lazy-mirror does not honor npm proxy settings (npm config) or http_proxy/https_proxy environment variables.

Either use the defaults (from npm or env variables) or allow a configuration section for proxy configuration.

Best way to add package in the cache only(not in npm)

Hello, me again.

In a future use, I would see a feature like:

Adding a package trough my cache and only on mine, not in npm open database.

ie a kind of private npm registry.

Probably can contribute to the feature, depending on my time, amount of work and Jdpaton wish for this software.
I'm not really an old JS dev, but i have very useful advice around me.

Jdpaton and all others, if you have opinions or advices, please share it.

Lucas