philips-software / npm-dependencies-extractor

Retrieves the (flat) list of package dependencies for a Javascript project using the npm package manager.

⚠️ THIS REPOSITORY IS NO LONGER MAINTAINED ⚠️

This is a CLI package that provides a command extract-dependencies to extract the flat list of (all installed) dependencies from a package-lock.json file (or another file with the same structure of your choice. If you want, instead of the package-lock.json, you may specify as input file the JSON file generated by the npm command to list json dependencies, such as:

npm list --json > inputFile.json

or, if you only want to see the production dependencies, generate the input as:

npm list --json --prod > inputFile.json 

Why the need to create npm-dependencies-extractor? The dependencies information as output by the package-lock.json or the npm list command is verbose, and requires some processing if you simply want to retrieve a (flat) list of your dependencies. Also, package-lock.json may contain more times a dependency that is present in different branches of the dependency tree.

When there is no dependency in your input file, the script exits after notifying the user. When your input does have dependencies, npm-dependencies-extractor generates, from your input, the flat list of dependencies, in two formats. The first output format is a js file containing the array of dependencies formatted as JSON objects (with keys name and version), with no nesting, like:

    [
        { ‘name’: ‘dependencyName_A’, ‘version’: ‘0.0.1’},
        { ‘name’: ‘dependencyName_B’, ‘version’: ‘2.0.0’},
        { ‘name’: ‘dependencyName_B’, ‘version’: ‘2.0.1’}
    ]

The second output format is a txt file containing an array of dependencies, one per line, formatted as name@version, like:

  [email protected]
  [email protected]
  [email protected]

Status

0.0.6, see CHANGELOG.md

Technology stack

• Javascript
• This software is intended to be used standalone, as a command-line tool

Prerequisites

• you should have Node installed (this script was tested with node v8.12.0)

• make sure you do not have a fixed depth configured in npm, so that npm install or npm list returns all dependencies (dependencies at all levels of depth).

npm config get depth

If something else than Infinity is returned, then remove the depth limitation by:

npm config delete depth

• you should create the input json with dependencies by either: (generates package-lock.json, with both dev and prod dependencies:)

npm install

or (generates a file similar to the structure of package-lock.json, but you may control whether to only contain dev or prod dependencies:)

npm list --json --prod > inputFile.json

• The following encodings of the input file are supported: utf8, utf16le. If the input file does not have a header containing the byte order mark, then you need to provide the encoding parameter, else the encoding is assumed to be utf8.

Installation

Install globally:

npm install -g npm-dependencies-extractor

Or you could use it without installing by running:

npx npm-dependencies-extractor [options]

Usage

extract-dependencies [options]

Supported options:

Flag	Alias	Functionality
--input [filename]	-i	Filename of the package-lock.json file to extract dependencies from. Default value: package-lock.json
--encoding	-e	(optional) Encoding of the input file. Allowed values: utf8, utf16le.
--optionals	-io	(optional) include optional dependencies
--devdependencies	-d	(optional) ignore dev dependencies
--output [filename]	-o	Js filename to which the flat list of dependencies is written. If the file already exists, it will be overwritten. Default value: dependencies.js. One more representation of the flat dependencies is generated, in the form of text (as .txt)
--verbose		Verbose output of commands and errors
--help	-h	Display help for command

Sample usage

npm run extract-dependencies -- -i ./test-data/input-with-optionals/package-lock-with-2-mandatory-dependencies.json --verbose

Usage scenarios

Scenario 1: You run the npm-dependencies-extractor's command without adding it as a dependency to your project

From the installation folder of npm-dependencies-extractor, run:

npm run extract-dependencies -- [options]

or, if you don't want to install it, run:

npx npm-dependencies-extractor [options]

or, if you don't want to install it from github master, run:

npx github:philips-software/npm-dependencies-extractor [options]

Scenario 2: You include the npm-dependencies-extractor as a dependency of your project, and call its command in your project's scripts, by:

extract-dependencies [options] 

FAQ

Question: I get an 'Unexpected token' error when my input JSON file is read; why?

Answer: This is most likely caused because your file is encoded in a format not supported yet, or because your file format is supported but its header does not contain a byte order mark (BOM) to describe its encoding. In the latter case, please provide to the script the encoding known by you by means of an additional parameter, like: --encoding <encodingOfTheInputFile>.

Currenlty supported values for encoding are: utf8, utf16le

Owners

See CODEOWNERS

Maintainers

See MAINTAINERS.md

Contributing

See CONTRIBUTING.md

License

See LICENSE.md

Author

Sanda Contiu

Keywords

• dependencies
• npm
• sbom
• software bill of material
• flat list
• extract
• retrieve
• dependencies flat list
• extract dependencies
• list dependencies

npm-dependencies-extractor's Issues

Migrate to Github Actions

Github Actions can preform all the necessary CI/CD steps.

Add travis-ci build pipeline

We've got a gitlab-ci build pipeline. Please make sure to port this to travis-ci

Add parameter to use the node_modules directory as source for the sBOM

We feel that sometimes it's hard to figure out what modules are used in the code.

In the package-lock.json file there might be some optional packages, f.e. fsevent has abbrev as optional dependency.

Talking to a frontend expert we've figured out, that we might use the node_modules directory to determine the sBOM.

Maybe we can add an argument to use the node_module as input. Suggestion argument: useNodeModulesDirectoryOnly

Publish npm package to npmjs.org

Good to get the license information also - report can be used as SOUP list

Extracting the npm packages with license details helps in analysing the SOUP

Make it an option to skip the optional dependencies from package-lock.json

Altering the behavior specified at #17 by making it optional to skip optional dependencies

This is because optional dependencies might still be installed (depending on e.g. the operating system that the dependency supports), and we might want to be on the safe side on reporting the sbom including those optional dependencies as well.

Remove `optional` dependencies from the list

Remove the optional dependencies from the list. The optional dependencies are only installed if they are required by another dependency. Ignoring them in the initial list will still result in a correct sBOM.

Example: js-react-app with dependency: fsevent. It has both abbrev and ansi-regex as optional. abbrev is not downloaded because it's not used by another dependency. ansi-regex is downloaded because it's also used by: strip-ansi. Filtering the optional dependencies will result in the correct sBOM: ansi-regex will be in and abbrev will not be there.

Support yarn.lock files as input.

Yarn projects are not supported. The tool can only read package-lock.json and npm-shrinkwrap.json files.

You can convert yarn.lock to package-lock.json with synp, but that does not cover optionals and devDependencies very well.

We need the yarn.lock to be used as input. Info on optionals are available as: optionalDependencies. devDependencies should be derived by combining info in the package.json (which can be a challenge in mono-repos with multiple package.jsons)

add flag to optionally create two additional outputs, for dev- and prod- only dependencies

Optionally generate additional output for

• development only dependencies, meaning, dependencies marked as "dev": true
• production only dependencies (non-dev), meaning, dependencies marked as "dev": false