meir017 / node-tgz-downloader

Downloads all of the node_modules tgz files based on a package-lock.json / pacakge.json file

node-tgz-downloader

Downloads all of the tarballs based on one of the following:

• local package-lock.json file
• url to a package-lock.json
• name of package
• local package.json file
• url to a package.json
• search keyword

install

npm install node-tgz-downloader -g

usage

From Code:

const downloader = require('node-tgz-downloader');

downloader.downloadFromPackageLock('path/to/package-lock');

From Command Line:

package-lock.json

from local file:

download-tgz package-lock path/to/package-lock.json

from url:

download-tgz package-lock https://raw.githubusercontent.com/Meir017/node-tgz-downloader/master/package-lock.json

package name

download-tgz package @angular/cli --devDependencies --peerDependencies

package.json

from local file:

download-tgz package-json path/to/package.json

from url:

download-tgz package-json https://raw.githubusercontent.com/Meir017/node-tgz-downloader/master/package.json

search keyword

downloads the packages returned from an npm search query (https://registry.npmjs.org/-/v1/search?)

download-tgz search tgz

HttpsAgent Configuration

The tool supports the https_proxy environment variable. Do not forget to specify protocol and port.

export https_proxy='https://proxy_url:port'

node-tgz-downloader's Issues

Download Rate Limiting

Possible to limit download concurrency (e.g. using p-limit) as I suspect I'm hitting the rate-limit for npm registry? I need to run several times (as previously downloaded tgz are skipped) before the download ran to completion.

Tried monkey-patching the original download

node-tgz-downloader/lib/downloader.js

Lines 52 to 58 in 9c144bf

	const promises = tarballs.map(({ url, directory }, i, arr) => {
	const position = `${i + 1}/${arr.length}`;
	logger(['downloading'.cyan, position], url);
	return _downloadFileWithRetry(url, join(baseDirectory, directory), position, 10);
	});
	return Promise.all(promises);

with the following and it worked flawlessly.

  const asyncSequentialDownload = async () => {
    for (let i = 0; i < tarballs.length; i++) {
      const { url, directory } = tarballs[i];
      const position = `${i + 1}/${tarballs.length}`;
      logger(['downloading'.cyan, position], url);
      await _downloadFileWithRetry(url, join(baseDirectory, directory), position, 10);
    }
  }

  return asyncSequentialDownload();

Failed to download

Hello when I user this command, I can' download the tgz。
download-tgz package-lock ./package-lock.json

[failed download tgz] undefined https://registry.npm.taobao.org/@vue/babel-helper-vue-jsx-merge-props/download/@vue/babel-helper-vue-jsx-merge-props-1.2.1.tgz?cache=0&sync_timestamp=1602851122331&other_urls=https%3A%2F%2Fregistry.npm.taobao.org%2F%40vue%2Fbabel-helper-vue-jsx-merge-props%2Fdownload%2F%40vue%2Fbabel-helper-vue-jsx-merge-props-1.2.1.tgz 10
[failed download tgz] undefined https://registry.npm.taobao.org/arr-diff/download/arr-diff-4.0.0.tgz 10

Every package is located in a different folder

Hi, I am using the downloader to pack all packages and dependencies, and upload them to a private NPM repository. It would be much easier to do so if the downloader would put all the tgz files in the same folder, instead of a different folder for each file.
Is it possible to apply this behaviour as an optional argument to the downloader?

Tarball directory structure to mirror NPM registry

Is it possible to allow the tarballs download directory structure to mirror npm registry? Specifically the .tgz would be in a - directory (e.g. node-tgz-downloader/-/node-tgz-downloader-3.11.2-preview1.tgz)?

The intent is for mirroring npm packages into AirGap environment, something like package-bundle, however package-bundle does not allow you to download packages according to package-lock.json file.

SyntaxError: Unexpected token '?'

_enumerateDependencies(tarballs, packageLock.packages ?? packageLock.dependencies, options);

Create http module

in order to simplify using a proxy we should move all http related function to a seperate module
#12

Error download version of package

Hi. for version of package show error.

download-tgz package @tensorflow/[email protected]

Add a configuration option for a different npm registry through command line

Not working with react-scripts

Hello,

I've tried your tool to download react-scripts package and the tool freeze at some point... Any idea ?

download-tgz package react-scripts

Regards,

proxy

Not working with proxy server, proxy wrote in .npmrc

download package.json failed

• macos 10.13
• node -v v16.14.2

when I run this command download-tgz package-json ./package.json --directory ./test --registry https://registry.npmjs.org/, it run successfully. but it seems the https turn to http in zsh,and there is no tarballs directory in the ./test directory in the end.

download-tgz package-lock is ok