strongloop / strong-build

Build node packages into deployable applications

strong-build

Build a node application package, preparing it for deploy to production.

It is useful standalone, but is commonly used to build applications for deployment to the StrongLoop process manager, strong-pm.

For more details, see http://strong-pm.io.

Installation

sl-build is made available through the strongloop tool as slc build, and works well with the StrongLoop Process Manager, strong-pm.

sl-build can be installed standalone with:

npm install -g strong-build

Overview

The purpose of building a node application is to bundle its dependencies so that there:

• are no deploy-time dependencies on external services
• it is in a deployable format

The build process is implemented as four commands:

• sl-build --install: the core of the build, it installs dependencies, runs custom build steps, and prunes development dependencies
• sl-build --bundle: modify the npm package.json and .npmignore configuration files so dependencies will be packed
• sl-build --pack: create an npm package of the build
• sl-build --commit: commit the build onto a git branch

The default behaviour of sl-build depends on whether the current directory is a git repository.

• In a git repository: the default is sl-build --install --commit, to build onto a git branch.
• Otherwise: the default is sl-build --bundle --install --pack, to build an npm package.

Both npm packages and git branches can be deployed to the StrongLoop process manager using the StrongLoop deploy tool.

Specifying any command disables the default, and allows any mix of commands to be run, either singly, or all at once.

Also, note that builds should be done in a clean working copy. You don't build deployment artifacts out of a possibly dirty working copy if you want reproducible builds. You can clean your working copy using git clean -x -d -f. This is too destructive for the build tool to do, but doing a build in an unclean working repository may trigger an error in a future version of the tool.

install command

Installation automates the common work flow for building application dependencies:

• npm install --ignore-scripts: Install dependencies without running scripts. Scripts can be run optionally with --scripts.
• npm run build: Custom build steps such as grunt build or bower can be specified in the package's scripts.build property, since front-end code served by node commonly requires some amount of preparation.
• npm prune --production: Remove development-only tools (such as bower, or grunt) that may have been required by the package's build scripts, but should not be deployed.

Note that compilation and install scripts should be run on the deployment server using:

• npm rebuild: Compile add-ons for current system.
• npm install: Run any install scripts (not typical, but if they exist they may be required to prepare the application for running).

If builds are done on the same system architecture as the deploy, it is possible to compile and package the add-ons, and avoid the presence of a compiler on the deployment system. This is recommended when possible, but is not the default assumption of strong-build.

bundle command

Bundling configures the package.json and .npmignore so deployment (not development) dependencies as well as any 'build' script output will not be ignored by npm pack.

This is unnecessary when using git to deploy, but mandatory when creating npm packages!

package.bundleDependencies

Bundling requires that the bundleDependencies property in the package.json file is configured to include all non-development dependencies, including optional dependencies, which are often overlooked.

Its important that you remember to add every new production dependency to the bundleDependencies property, if you don't, npm will try and install them after deploy, creating unexpected and fragile dependencies on npmjs.org.

Since keeping this up-to-date manually is likely to go wrong, we recommend allowing the bundle command to do this for you. However, the bundleDependencies property is not modified if present, so you are free to maintain it yourself, if you wish (or to not just the bundle command).

.npmignore

Setting bundle dependencies is insufficient to get the output of build tools.

Both build output and project ephemera such as test output is usually ignored using .gitignore, as it should be. However, if npm does not find a .npmignore configuration file, it uses .gitignore as a fallback. This means that if you have custom build output, such as minimized JavaScript, it will be treated as project ephemera, and not be packed by npm.

The bundle command will create an empty .npmignore file if there is a .gitignore file but there is no .npmignore file. This will work for clean repositories, but if you have any project ephemera, they will get packed.

We recommend you write and maintain you own .npmignore file unless your build process guarantees a clean working repository.

pack command

Pack output is a tar file in the format produced by npm pack and accepted by npm install and strong-deploy.

The pack file is placed in the parent directory of the application being packed, to avoid the pack file itself getting packed by future builds, and to allow the working repository to be cleaned.

If a .npmignore file was created by the bundle command, check the pack file contents carefully to ensure build products are packed, but project ephemera are not.

commit command

Committing build products into git provides the most robust tracking and storage, including versioning of deployments.

This is often done by committing both build products and dependencies (node_modules) into git where they pollute the source branches, create massive git commits and huge churn on the development branches and repositories.

The commit command does not do this.

It commits an exact replica of current branch source and build products onto a deployment branch. After the commit, the deployment branch tip shows as a merge of the deployment and source branches. This allows a complete history of deployment builds to be kept in git, but separated from the development branches. Deployment branches can be pushed to the same repository as the development branches, or not.

Note that branches prepared like this can also be pushed to platforms such as OpenShift and Heroku.

The default name of the deployment branch is "deploy", but is configurable with the --onto BRANCH modifier to the commit command.

Usage

usage: sl-build [options]

Build a node application package.

With no options, the default depends on whether a git repository is
detected or not.

If a git repository is detected the default is `--git`: install and commit the
build results to the "deploy" branch, which will be created if it does not
already exist.

If no git repository is detected the default is `--npm`: bundle, install, and
pack the build results into a `<package-name>-<version>.tgz` file.

Options:
  -h,--help       Print this message and exit.
  -v,--version    Print version and exit.
  -n,--npm        Same as `--install --bundle --pack`.
  -g,--git        Same as `--install --commit`.
  -i,--install    Install dependencies (without scripts, by default).
  --scripts       If installing, run scripts (to build addons).
  -b,--bundle     Modify package to bundle deployment dependencies.
  -p,--pack       Pack into a publishable archive (with dependencies).

Git specific options:
  -c,--commit     Commit build output (branch specified by --onto).
  --onto BRANCH   Branch to commit build results to, created if
                  necessary ("deploy", by default).

License

strong-build uses a dual license model.

You may use this library under the terms of the Artistic 2.0 license, or under the terms of the StrongLoop Subscription Agreement.

strong-build's Issues

Update docs for --npm and --git options

PR #25 adds new options to the slc build command.

At a minimum, update the command ref.
Also review to see if any changes are needed in these (and child articles):

• http://docs.strongloop.com/display/SLC/Building+applications+with+slc
• http://docs.strongloop.com/display/SLC/Deployment+best+practices

NOTE: We should not make the doc changes until the change is published to npm.

tarball packed by slc build -ibp --scripts does not include the build/ subdirectory

I did

cd /tmp/
npm install statsd
cd node_modules/statsd
slc build -ibp --scripts
tar -tf ../statsd-0.7.2.tar.gz | grep build  # nothing!
find . -print | grep build  # lots!

And the packfile produced did not include node_modules/node-syslog/build/....

An -o to specify where the archive should be output.

Being able to specify where the build archive should go would be sweet. Just defaulting to the parent dir without any option to specify otherwise is jank.

Can we have the --no-optional flag?

Can we add a --no-optional so slc build will install dependencies with npm install --no-optional? Our company's policy doesn't allow internet access on the build server, which slows down the build a lot when it tries to install an optional dependency sl-blip. Thanks!

slc build --install should force a reinstall of deps.

I've been running into an issue where every second time I run this command slc build -ibp the npm run build command, which is automatically invoked, fails. I get the following output:

$ slc build -ibp --scripts
Running `npm install`
Running `npm run build`
Error on `npm run build`:
Failed to run `npm run build`:

> <my project>
> gulp default


Error: Cannot find module '<path_to_my_project>/node_modules/gulp/node_modules/v8flags/3.14.5.9.flags.json'

I know this looks like an issue with v8flags, but I'm not so sure that it is. When I rm -rf my node_modules and then do a fresh npm install I can invoke npm run build to my hearts content; but, when I invoke slc build -ibp, it will work only the first time.

Any thoughts??

Bundle private / scoped dependencies with slc build --npm

Hello,

I'm not sure if this issue belongs here or within npm, but I'll start here and see if there is a solution within strongloop / slc.

I am using slc build --npm to package my node app into a .tgz that I can slc deploy to my production web server. This was working well when all of my dependencies were public npm modules.

Recently I've started using private npm modules that are scoped and hosted on nppmjs.org (https://www.npmjs.com/private-modules)

slc build -b appropriately adds my scoped module (e.g. @barwin/my-private-module) to the bundleDependencies array in package.json, but slc build --npm does not ultimately include the scoped module in the resulting .tgz file. All of my non-scoped (public) npm dependencies are included in the .tgz as expected. It's only an issue with the scoped modules.

I found a snippet in npm's faq about scoped dependencies:

https://docs.npmjs.com/misc/faq
Unscoped packages can only depend on other unscoped packages. Scoped packages can depend on packages from their own scope, a different scope, or the public registry (unscoped).

Based on that, I tried renaming my top level project to be under the same scope as the private modules that I'm trying to bundle, but that had no effect.

Looking for any suggestions on how to accomplish this. Thanks!

`Slc build --scripts` should run the default commands

In a non-git archive, it does nothing.

'npm run build' fails if there is no build script with npm 2

In npm@^1.0.0 npm run X is a no-op that exits with 0 if X is not a script in package.json.
This behaviour has long been considered a bug (npm/npm#2959) and was finally "fixed" in 2.1.8 via npm/npm@0aefae9.

The result is slc build will fail if the user is running the latest npm if they have not defined a build script in package.json.

strong-build tests failing on io.js

See #31 (comment)

find workaround for warnings about CRLF to LF conversions

http://stackoverflow.com/questions/1967370/git-replacing-lf-with-crlf

core.autocrlf set to false guarantees what gets checked out is what got checked in, this is appropriate for our use case

as-is, since npm (bless it) uses UNIX conventions for most of the files it writes, including the ones from most package.tgz files from npmjs.org, the "warning: LF will be replaced by CRLF in ...." will be spewed endlessly to the screen.... for files that the user has no control over.

We have to make sure its only temporarily false, though, and not mess with the user's defaults.

git not detected

Running `mv -f testappfive-0.0.0.tgz ../testappfive-0.0.0.tgz`
Seans-MacBook-Air-2:testappfive seanbrookes$ slc deploy http://:8765
URL `http://:8765` is not valid
Seans-MacBook-Air-2:testappfive seanbrookes$ slc deploy http://localhost:8765
URL `http://localhost:8765` is not valid
Seans-MacBook-Air-2:testappfive seanbrookes$ sl-deploy http://localhost:8765
URL `http://localhost:8765` is not valid
Seans-MacBook-Air-2:testappfive seanbrookes$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   package.json

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .editorconfig
        .gitignore
        .jshintignore
        .jshintrc
        .npmignore

no changes added to commit (use "git add" and/or "git commit -a")
Seans-MacBook-Air-2:testappfive seanbrookes$ slc build
Running `npm install --ignore-scripts`
Running `npm run build`
Running `npm prune --production`
Running `npm --quiet pack`
Running `mv -f testappfive-0.0.0.tgz ../testappfive-0.0.0.tgz`
Seans-MacBook-Air-2:testappfive seanbrookes$ git rev-parse --git-dir
.git
Seans-MacBook-Air-2:testappfive seanbrookes$  

Fails when app has 0 deps

Here is my app:

require('net').createServer().listen(0);

Here is what I get from slc build

Running `npm install --ignore-scripts`
Running `npm prune --production`
shell.js: internal error
Error: ENOENT, no such file or directory 'node_modules'
    at Error (native)
    at Object.fs.statSync (fs.js:801:18)
    at /usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/find.js:42:12
    at Array.forEach (native)
    at Object._find (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/find.js:39:9)
    at Object.find (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/shelljs/src/common.js:186:23)
    at Object.doNpmPack [as func] (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/index.js:290:29)
    at Immediate._onImmediate (/usr/local/lib/node_modules/strongloop/node_modules/strong-build/node_modules/vasync/lib/vasync.js:213:20)
    at processImmediate [as _immediateCallback] (timers.js:358:17)

@kraman

Support .npmignore when committing to deploy branch in git?

It seems that the .npmignore file is only observed by the --pack option, not the --commit option. I woud prefer to use git as the mechanism to store our versioned builds, but obviously the git repository for my project contains a test folder for my tests. If I add 'test' to .npmignore and pack a build into tgz then the test folder (and anything else in .npmignore) is not included... alas if I commit a build to a deployment branch and then git clone/pull from this branch on the production machine, the test folder comes along for the ride.

I suppose there is perhaps no harm in these things hanging around on the production machine, but I'd prefer that they weren't there. (This also goes for things like IDE project files which are committed to git, but I would also add to a .npmignore file to have them excluded from the package or deploy branch commit).

Native modules?

Curious how does string-build deal with native modules? Does this also mean that build machine has to be identical (or pretty close) in configuration to production?