larsw / npm-leech

Small utility to leech all direct and transitive npm packages for a given package.json or package-lock.json file.

npm-leech

Small utility to leech all direct and transitive npm packages for a given package.json or package-lock.json file.

Ideal for mirroring a given set of packages to a private repository like JFrog Artifactory or NPM Registry.

    usage
      $ npm-leech [-i package.json|package-lock.json] [-o foo.tar] [-a] [-c] [-d] [-D] 

    options
      --artifactory, -a      Store scoped packages in a layout used by Artifactory
      --input, -i            source package.json or package-lock.json (default: ./package-lock.json)
      --output, -o           target tarballs tar (default: ./npm-tarballs.tar)
      --concurrency, -c      number of concurrent retrieval tasks for meta/pkg (default: 4)
      --dev, -d              leech devDependencies in source. (default: false)
      --transitive-dev, -D   CAUTION! leech all transitive devDependencies. (default: false)
      --registry, -r         NPM registry. (default: http://registry.npmjs.org/)
      --verbose, -v          Verbose output. (default: false)
      --progress, -p         Progress bar. Should not be used with -v (default: true)

    examples
      $ npm-leech -i ../../package.json -o foo.tar -c 8 -d

License

MIT - @larsw

npm-leech's Issues

Option to store scoped packages in Artifactory layout

Hi,
In he npmjs.org registry scoped packages has a tarball path like this:

# npm info '@types/qs@latest' --json dist.tarball
"https://registry.npmjs.org/@types/qs/-/qs-6.9.6.tgz"

I.e. "@scope/packagename/-/packagename-version.tgz"
When downloaded by npm-leech, the file structure in the tar file is:

# tar tvf npm-tarballs.tar 
-rw-r--r-- 0/0            2405 2021-05-05 18:23 @types/qs-6.9.6.tgz

When this tar-file is imported to Artifactory using the web-interface, this layout from the tar file is kept in Artifactory. This is not the "correct" path inside Artifactory, it uses a path like this for scoped packages:

@scope/packagename/~/@scope/packagename-version.tgz

When importing the tar file from npm-leech into Artifactory, the incorrect layout will cause issues with scoped packages that also exist without a scope with the same name and version. Like "@types/qs" and "qs". It will cause checksum error when trying to install (npm install) a package like this using Artifactory as a registry. See this bug for more info:
https://www.jfrog.com/jira/browse/RTFACT-7668

Several issues like this is reported for Artifactory, but no response from the developer that I could see.

Pushing the packages to Artifactory using "npm publish" will go through the Artifactory API and store the file with the layout expected by Artifactory.

But I hoped this could be fixed in npm-leech by adding a option to store in Artifactory layout inside the tar file. This way the bulk import of the tar file from Artifactory web interface would work out of the box.

Convert to TypeScript

CI/CD

• GitHub actions build/test/publish to NPM

npm-leech do not download dependencies with tilde (~) and caret (^) in versions

Hi,
While using npm-leech to download a package with a lot of dependencies, some dependencies failed to download with an error message like this:

pkg-err { Error: Request failed with status code 405
    at createError (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/usr/lib/node_modules/npm-leech/node_modules/axios/lib/adapters/http.js:236:11)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:978:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  config: 
   { url: 'http://registry.npmjs.org/accepts/~1.3.7',
...
     _header: 'GET /accepts/~1.3.7 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.19.2\r\nHost: registry.npmjs.org\r\nConnection: close\r\n\r\n',
...
        responseUrl: 'http://registry.npmjs.org/accepts/~1.3.7',
        redirects: [],
        read: [Function] } },
  response: 
   { status: 405,
     statusText: 'Method Not Allowed',

The package.json used with npm-leech version 1.2.1:

  "name": "foo",
  "version:": "1.0.0",
  "dependencies": {
    "express": "latest"
  }
}

The package 'express' dependencies, only showing the two first:

npm show express --json dependencies
{
  "accepts": "~1.3.7",
  "array-flatten": "1.1.1",

The dependency 'array-flatten' has a "normal" semver and is downloaded without problems.
But the dependency 'accepts' has a tihlde in the semver. This causes npm-leech to download this version without resolving the tihlde to a "normal" semver first. But the registry does not understand "GET http://registry.npmjs.org/accepts/~1.3.7".

Using wireshark, this is the TCP stream when downloading the package 'accepts':

GET /accepts/~1.3.7 HTTP/1.1
Accept: application/json, text/plain, */*
User-Agent: axios/0.19.2
Host: registry.npmjs.org
Connection: close

HTTP/1.1 405 Method Not Allowed
Date: Wed, 05 May 2021 15:47:03 GMT
Content-Type: application/json
Content-Length: 63
Connection: close
Set-Cookie: __cfduid=db421dde183435235cd4cb62992f582121620229622; expires=Fri, 04-Jun-21 15:47:02 GMT; path=/; domain=.npmjs.org; HttpOnly; SameSite=Lax
CF-Ray: 64ab1ae7687ffac0-OSL
Allow: PUT
CF-Cache-Status: DYNAMIC
cf-request-id: 09decf24a30000fac0cd049000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare

{"code":"MethodNotAllowedError","message":"GET is not allowed"}

So, shouldn't npm-leech resolve tihlde and caret semvers into normal semvers before downloading packages?