bestander / npm-git-lock

Easy and reliable alternative to shrinkwrap or committing node_modules

npm-git-lock

A CLI tool to lock all node_modules dependencies to a separate git repository.

Read a post why you may need it.

Update

I npm-git-lock was created a few years ago before Yarn and offline mirror feature: https://yarnpkg.com/blog/2016/11/24/offline-mirror/.

There is even a feature to store built artifacts, so I would suggest switching to Yarn as a more scalable solution.

Features

• Tracks changes in package.json file
• When a change is found makes a clean install of all dependencies and commits and pushes node_modules to a remote repository
• Works independently from your npm workflow and may be used on a CI server only keeping your dev environment simpler

How to use

sudo npm install -g npm-git-lock
cd [your work directory]  
npm-git-lock --repo [[email protected]:your/dedicated/node_modules/git/repository.git] -v

If you don't want to depend on NPM connectivity when installing this module, you can install directly from github:

sudo npm install -g https://raw.githubusercontent.com/bestander/npm-git-lock/master/npm-git-lock-latest.tgz

• Beware of possible breaking changes in the future, if you seek stability, obtain a link to a particular commit with the .tgz file on GitHub.

Options:

--verbose                 [-v] Print progress log messages
--repo                    Git URL to repository with node_modules content  [required]
--cross-platform          Run in cross-platform mode (npm 3 only)
--incremental-install     Keep previous modules instead of always performing a fresh npm install (npm 3 only)
--production              Runs npm install with production flag
--check-all-json-elements Sha-1 calculated from all elements in package.json instead of only dependencies and devDependencies

npm-git-lock works with both npm 2 and 3, although the options --cross-platform and --incremental-install are only supported on npm 3.

Why you need it

You need it to get reliable and reproducible builds of your Node.js/io.js projects.

Shrinkwrapping

is the recommended option to "lock down" dependency tree of your application.
I have been using it throughout 2014 and there are too many inconveniences that accompany this technique:

1. Dependency on npm servers availability at every CI build. NPM availability is quite good in 2015 watch a good talk but you don't want to have another moving part when doing an urgent production build.
2. Managing of npm-shrinkwrap.json is not straightforward as of [email protected]. It is promising to improve though.
3. Even though npm does not allow force updating packages without changing the version, packages can still be removed from the repository and you don't want to find that out when doing a production deployment.
4. There are lots of other complex variable things about shrinkwrapping like optional dependencies and the coming changes in [email protected] like flat node_modules folder structure.

Committing packages

to your source version control system was recommended before shrinkwrapping but it is not anymore.
Nonetheless I think it is a more reliable option though with a few annoying details:

1. A change in any dependency can generate a humongous commit diff which may get your Pull Requests unreadable
2. node_modules often contains binary dependencies which are platform specific and large and don't play well across dev and CI environments

npm-git-lock is like committing your dependencies to git but without the above disadvantages.

How it works

The algorithm is simple:

1. Check if node_modules folder is present in the working directory
2. If node_modules exists check if there is a separate git repository in it
3. Calculate sha1 hash from package.json in base64 format
4. If remote repo from [2] has a commit tagged with sha1 from [3] then check it out clean, no npm install is required
5. Otherwise remove everything from node_modules (unless --incremental-install is set, in which case only uncommitted changes will be stashed away), do a clean npm install, commit, tag with sha1 from [3] and push to remote repo
6. Next time you build with the same package.json, it is guaranteed that you get node_modules from the first run

After this you end up with a reliable and reproducible source controlled node_modules folder.
If there is any change in package.json, a fresh npm install will be done once.
If there is no change, npm command is not touched and your CI build is fast.

Cross-platform mode

When npm-git-lock is run with the --cross-platform option, it does not commit "platform-specific" build artifacts into the remote repository. Instead, it builds them using npm rebuild when checking out the repository (step 4) or when doing a clean npm install (step 5). Platform-specific files are taken to be those files that are generated by build scripts.

Inspired by this post, this is how step 5 is modified in cross-platform mode:

1. Run npm install --ignore-scripts to prevent platform-specific compilation of any files.
2. Run git add . to capture the current "clean" cross-platform state.
3. Run npm rebuild to create any platform-specific files.
4. Run git status --untracked-files=all to list all files that have been generated in the previous step. Add these files to .gitignore.

--cross-platform is only supported on npm version >= 3, since npm 2 doesn't run custom install scripts as it should during npm rebuild (cf. this CI failure).

Incremental installs

By default, npm-git-lock will perform a completely fresh npm install whenever there is any change to package.json (i.e., there is no commit in the node_modules repository tagged with the sha1 of package.json). However, that might not always be desired, since all dependencies might change (as long as their version is still within the range specified in package.json).

To get a behavior more similar to npm shrinkwrap, you can use the option --incremental-install. When installing modules, it will reuse modules that have already been committed to the node_modules repository and only run npm install "on top of them".

A potential caveat is that modules are always fetched from the latest state (the master branch) of the node_modules repository. If there are dependencies introduced in a previous commit but not on the latest master HEAD, they will be freshly installed.

Example: In your project, you work on a branch A that declares a new dependency depA in package.json. In parallel, you have a branch B declaring a new dependency depB. Then you merge them both back into master. Assuming you run npm-git-lock in each step, the history of your central node_modules repository will look like this (from recent to older):

• [commit3, master HEAD] depA + depB
• [commit2] depB
• [commit1] depA

Note that when running npm-git-lock after the merge (to produce commit3), only depB was fetched from the previous state. For depA, a fresh install was performed.

Amazing features

With this package you get:

1. Minimum dependency on npm servers availability for repeated builds which is very common for CI systems.
2. No noise in your main project Pull Requests, all packages are committed to a separate git repository that does not need to be reviewed or maintained.
3. If the separate git repository for packages gets too large and slows down your builds after a few years, you can just create a new one, saving the old one for patches if you need.
4. Using it does not interfere with the recommended npm workflow, you can use it only on your CI system with no side effects for your dev environment or mix it with shrinkwrapping.
5. You can have different node_modules repositories for different OS. Your CI is likely to be linux while your dev machines may be mac or windows. You can set up 3 repositories for them and use them independently.
6. And it is blazing fast.

Troubleshoot

If you see this kind of error in your CI:

Cloning into 'node_modules'...
done.
*** Please tell me who you are.
Run
  git config --global user.email "[email protected]"
  git config --global user.name "Your Name"
to set your account's default identity.
Omit --global to set the identity only in this repository.
fatal: empty ident name (for <[e-mail]>) not allowed

You need to configure user.email and user.name in the environment as shown in the error message.
Just add those two commands before npm-git-lock call.

If you see this kind of error:

fatal: bad revision 'HEAD'
fatal: bad revision 'HEAD'
fatal: Needed a single revision
You do not have the initial commit yet

You need to commit and push to the repote repository at least once before using npm-git-lock

If you see this kind of error:

Git command 'tag -l --points-at HEAD' failed:
error: unknown option `points-at'

or

Git command 'stash save --include-untracked' failed:
error: unknown option for 'stash save': --include-untracked

You need to upgrade git to version 1.7.10+.

Contribution

Please give me your feedback and send Pull Requests.
Unit tests rely on require(`child_process`).execSync command that works in node 0.11+.

Future plans (up for grabs)

• Replace .es6 extension with .js
• Switch to shelljs from promises API. Promises are still too heavy for such a file oriented CLI tool

Change Log

3.6.0 - 2018-02-20

• Feature fixed scoped packages for --cross-platform flag

3.5.0 - 2016-06-30

• Feature support --check-all-json-elements

3.3.0 - 2016-04-26

• Feature support --production

3.2.1 - 2016-04-14

• Fixed support for Node 0.12

3.2.0 - 2016-03-24

• Feature run preinstall and postinstall scripts even in --cross-platform mode

3.1.1 - 2016-03-17

• Fixed loglevel argument for npm commands

3.0.0 - 2016-02-18

• The hashing algorithm has changed due to a bug that caused different hashes to be generated on different platforms. This means hashes generated by 3.0.0+ are not compatible with older versions. Make sure you use the same version in all your environments! (git install -g [email protected] is your friend)

License MIT

npm-git-lock's Issues

loglevel parameter is incorrect

The loglevel parameter passed to npm in npmRunCommands() in checkout-node-module.js should be "--loglevel" and not "--log-level" with a dash.

This is causing an issue for me because TFS Build treats anything written to STDERR as a build failure, which is where npm writes warnings. When I make this change locally, it properly silences npm warnings when the -v flag is absent.

BTW.. Thanks for this nice tool and comprehensive blog post!

Should --cross-platform run 'npm install'?

Hi there,

as title says, should --cross-platform run 'npm install'? I would guess it shouldn't, because some platform specific dependencies, such as dbus, are likely to fail as they aren't multi-platform.

My project is meant to run on a RPI(arm), my dev machine is MacOS and CI is Ubuntu Linux (x64).
When I run the command below on MacOS, it fails because some of my dependencies are RPI specific.

npm-git-lock --cross-platform --incremental-install --production --repo [my repo url] -v

Therefore, node_modules never gets push to its repor, despite being installable in the end users machine(RPI).

Thoughts?

add npm version and version from package.json to commit comment

Support yarn, pnpm and so on

Is it possible to use Yarn and pnpm with this?
Because it seems that there is currently an npm package to run npm scripts but we would like to use Yarn or pnpm instead.

Would be good to have some flag to change the used package manager.

Error with --cross-platform option and scoped packages on npm v5.x

With the option --cross-platform and npm v5.x while having a scoped package in the package.json, npm-git-lock tries to rebuild the scope folder instead of the packages contained in the scope.

For npm v3.x this was not a problem, as the rebuild command for the scope folder did nothing.
However, for npm v5.x, it throws the following exception (here with @angular/cli):

npm ERR! code EINVALIDTAGNAME
npm ERR! Invalid tag name "@angular": Tags may not have any characters that encodeURIComponent encodes.
npm command 'rebuild' failed:
npm ERR! code EINVALIDTAGNAME
npm ERR! Invalid tag name "@angular": Tags may not have any characters that encodeURIComponent encodes.

possible git commit conflicts

Checkout latest commit from master branch before removing all and making a new commit.
Here is a scenario:

1. create snapshot A and commit
2. create snapshot B with one file removed and commit
3. checkout snapshot A for a build
4. create snapshot C that derives from A but conflicts with B
5. git push will may throw an exception

Fails without an initial commit

Nice software. Thanks! Looks like either the instructions need expanded, or the code needs updated.

Without doing an initial commit, the git stash command fails. It looks like https://github.com/bestander/npm-git-lock/blob/master/src/checkout-node-modules.es6#L270 needs an error catch or a test against git rev-parse HEAD or git status (or similar) before running. This is with an empty remote repo and an empty node_modules directory.

james@server:~/test$ npm-git-lock --repo [email protected]:Xeboc/node.git -v
Updating /home/james/test/node_modules using repo [email protected]:Xeboc/node.git
Sha-1 calculated from dependencies and devDependencies in package.json.
SHA-1 of package.json (version 0.0.1) is RFpRkRTRMU0iS_512wiINKCimws=
Checking if remote [email protected]:Xeboc/node.git exists
Remote exists, fetching from it
Git command 'git fetch -t [email protected]:Xeboc/node.git' failed:
fatal: Couldn't find remote ref HEAD

Remote [email protected]:Xeboc/node.git is not present in /home/james/test/node_modules/.git repo
Removing /home/james/test/node_modules
Cloning [email protected]:Xeboc/node.git
Remote [email protected]:Xeboc/node.git is in node_modules, checking out RFpRkRTRMU0iS_512wiINKCimws= tag
Requested tag does not exist, installing node_modules
Git command 'stash save --include-untracked' failed:
fatal: bad revision 'HEAD'
fatal: bad revision 'HEAD'
fatal: Needed a single revision
You do not have the initial commit yet

Failed to synchronise node_modules with [email protected]:Xeboc/node.git: Error: 'git stash save --include-untracked' exited with error code 1

So I committed a file:

james@server:~/test/node_modules$ touch asdf
james@server:~/test/node_modules$ git add .
james@server:~/test/node_modules$ git commit -a -m "initial commit"
[master (root-commit) 84b7c65] initial commit
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 asdf
james@server:~/test/node_modules$ git push
Counting objects: 3, done.
Writing objects: 100% (3/3), 207 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To [email protected]:Xeboc/node.git
 * [new branch]      master -> master

And then success:

james@server:~/test$ npm-git-lock --repo [email protected]:Xeboc/node.git -v
Updating /home/james/test/node_modules using repo [email protected]:Xeboc/node.git
Sha-1 calculated from dependencies and devDependencies in package.json.
SHA-1 of package.json (version 0.0.1) is RFpRkRTRMU0iS_512wiINKCimws=
Checking if remote [email protected]:Xeboc/node.git exists
Remote exists, fetching from it
Remote [email protected]:Xeboc/node.git is in node_modules, checking out RFpRkRTRMU0iS_512wiINKCimws= tag
Requested tag does not exist, installing node_modules
Removing everything from node_modules
This might take a few minutes -- please be patient

Support for typings folder

Similar to the node_modules, it would be very interesting to also lock the typings folder, since it would also help in reducing external connection requirements.

`preinstall` and `postinstall`

I noticed that because you have to use --ignore-scripts when doing --cross-platform, that pre and post install scripts in my project are no longer run (which makes sense haha).

Thoughts on manually calling preinstall and postinstall if --cross-platform is set? Happy to submit a PR.

Releases after 2.1.8 are not compatible with older versions

The hash generated by 2.1.8 is different than the one generated by 3.x.x for the same package.json.
This being a npm install -g module one can easily forget to update it locally.
I know I can and should update my "setting up a dev environment" document to say npm install -g [email protected] but please add a big Changelog section somewhere in README.md making this clear :)

Great tool otherwise.

package.json different but node_modules the same except for supposed EOL changes

I have encountered some behavior that I'm not sure how to work around.

I have npm-git-lock included as part of my build process via the following 2 npm scripts:

"gitlock": "npm-git-lock --repo https://repo-url-here.git --verbose --cross-platform --incremental-install",
"start": "npm run gitlock && grunt"

The issue I have just encountered is that I updated package.json, and so per https://github.com/bestander/npm-git-lock#how-it-works, npm-git-lock begins with step 1 and goes through to step 5, where it fails at the commit I think because npm-git-lock tries to commit node_modules even though the node_modules content in the remote repo is actually identical to the node_modules content that npm-git-lock is trying to commit, hence there is nothing to commit. This seems to be because, the change to the package.json was just the addition of a package that was already previously installed and committed to the remote repo because it was a dependency of a package that was included in package.json at the time of a previous execution of the npm-git-lock command.

Since step 5 results in an error with exit code 1, my npm start script fails.

It looks like https://github.com/bestander/npm-git-lock/blob/master/src/checkout-node-modules.es6#L331 is already checking whether there are any local changes, and when I run git status in the git repo that npm-git-lock clones within node_modules, it does show some files have been modified, but the modifications are only end-of-line changes which disappear when the files are staged. So I believe gitHasChanges is returning true due to the EOL modification warnings, but then the commit ultimately fails due to the line ending modifications disappearing during staging....

Any advice on how to deal with this issue would be much appreciated!

Production only option?

As a user of this module I'd love to be able to supply a --production flag to have only production modules installed / tracked

use tags/<tag name> for checkout