isabella232 / wireit

Wireit upgrades your npm scripts to make them smarter and more efficient.

Wireit upgrades your npm scripts to make them smarter and more efficient.

Features

• 🙂 Use the npm run commands you already know
• ⛓️ Automatically run dependencies between npm scripts in parallel
• 👀 Watch any script and continuously re-run on changes
• 🥬 Skip scripts that are already fresh
• ♻️ Cache output locally and remotely on GitHub Actions for free
• 🛠️ Works with single packages, npm workspaces, and other monorepos

Alpha

🚧 Wireit is alpha software — in active but early development. You are welcome to try it out, but note there a number of missing features and issues that you may run into! 🚧

Contents

• Features
• Install
• Setup
• VSCode Extension
• Dependencies
  • Vanilla scripts
  • Cross-package dependencies
• Parallelism
• Extra arguments
• Input and output files
• Incremental build
• Caching
  • Local caching
  • GitHub Actions caching
• Cleaning output
• Watch mode
• Failures and errors
• Package locks
• Recipes
  • TypeScript
  • ESLint
• Reference
  • Configuration
  • Dependency syntax
  • Environment variables
  • Glob patterns
  • Fingerprint
• Requirements
• Related tools
• Contributing

Install

npm i -D wireit

Setup

Wireit works with npm run, it doesn't replace it. To configure an NPM script for Wireit, move the command into a new wireit section of your package.json, and replace the original script with the wireit command.

Before	After
{ "scripts": { "build": "tsc" } }	{ "scripts": { "build": "wireit" }, "wireit": { "build": { "command": "tsc" } } }

Now when you run npm run build, Wireit upgrades the script to be smarter and more efficient. Wireit works with yarn (both 1.X "Classic" and its successor "Berry") and pnpm, too.

You should also add .wireit to your .gitignore file. Wireit uses the .wireit directory to store caches and other data for your scripts.

echo .wireit >> .gitignore

VSCode Extension

If you use VSCode, consider installing the google.wireit extension. It adds documentation on hover, autocomplete, can diagnose a number of common mistakes, and even suggest a refactoring to convert an npm script to use wireit.

Install it from the marketplace or on the command line like:

code --install-extension google.wireit

Dependencies

To declare a dependency between two scripts, edit the wireit.<script>.dependencies list:

{
  "scripts": {
    "build": "wireit",
    "bundle": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc"
    },
    "bundle": {
      "command": "rollup -c",
      "dependencies": ["build"]
    }
  }
}

Now when you run npm run bundle, the build script will automatically run first.

Vanilla scripts

The scripts you depend on don't need to be configured for Wireit, they can be vanilla npm scripts. This lets you only use Wireit for some of your scripts, or to upgrade incrementally. Scripts that haven't been configured for Wireit are always safe to use as dependencies; they just won't be fully optimized.

Cross-package dependencies

Dependencies can refer to scripts in other npm packages by using a relative path with the syntax <relative-path>:<script-name>. All cross-package dependencies should start with a ".". Cross-package dependencies work well for npm workspaces, as well as in other kinds of monorepos.

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc",
      "dependencies": ["../other-package:build"]
    }
  }
}

Parallelism

Wireit will run scripts in parallel whenever it is safe to do so according to the dependency graph.

For example, in this diagram, the B and C scripts will run in parallel, while the A script won't start until both B and C finish.

graph TD
  A-->B;
  A-->C;
  subgraph parallel
    B;
    C;
  end

By default, Wireit will run up to 4 scripts in parallel for every CPU core detected on your system. To change this default, set the WIREIT_PARALLEL environment variable to a positive integer, or infinity to run without a limit. You may want to lower this number if you experience resource starvation in large builds. For example, to run only one script at a time:

export WIREIT_PARALLEL=1
npm run build

If two or more seperate npm run commands are run for the same Wireit script simultaneously, then only one instance will be allowed to run at a time, while the others wait their turn. This prevents coordination problems that can result in incorrect output files being produced. If output is set to an empty array, then this restriction is removed.

Extra arguments

As with plain npm scripts, you can pass extra arguments to a Wireit script by placing a -- double-dash argument in front of them. Any arguments after a -- are sent to the underlying command, instead of being interpreted as arguments to npm or Wireit:

npm run build -- --verbose

Input and output files

The files and output properties of wireit.<script> tell Wireit what your script's input and output files are, respectively. They should be arrays of glob patterns, where paths are interpreted relative to the package directory. They can be set on some, all, or none of your scripts.

Setting these properties allow you to use more features of Wireit:

	Requires files	Requires output
Dependency graph	-	-
Watch mode	☑️	-
Clean build	-	☑️
Incremental build	☑️	☑️
Caching	☑️	☑️

Example configuration

{
  "scripts": {
    "build": "wireit",
    "bundle": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc",
      "files": ["src/**/*.ts", "tsconfig.json"],
      "output": ["lib/**"]
    },
    "bundle": {
      "command": "rollup -c",
      "dependencies": ["build"],
      "files": ["rollup.config.json"],
      "output": ["dist/bundle.js"]
    }
  }
}

Incremental build

Wireit can automatically skip execution of a script if nothing has changed that would cause it to produce different output since the last time it ran. This is called incremental build.

To enable incremental build, configure the input and output files for each script by specifying glob patterns in the wireit.<script>.files and wireit.<script>.output arrays.

ℹ️ If a script doesn't have a files or output list defined at all, then it will always run, because Wireit doesn't know which files to check for changes. To tell Wireit it is safe to skip execution of a script that definitely has no input and/or files, set files and/or output to an empty array (files: [], output: []).

Caching

If a script has previously succeeded with the same configuration and input files, then Wireit can copy the output from a cache, instead of running the command. This can significantly improve build and test time.

To enable caching for a script, ensure you have defined both the files and output arrays.

ℹ️ If a script doesn't produce any output files, it can still be cached by setting output to an empty array ("output": []). Empty output is common for tests, and is useful because it allows you to skip running tests if they previously passed with the exact same inputs.

Local caching

In local mode, Wireit caches output files to the .wireit folder inside each of your packages.

Local caching is enabled by default, unless the CI=true environment variable is detected. To force local caching, set WIREIT_CACHE=local. To disable local caching, set WIREIT_CACHE=none.

⚠️ Wireit does not currently limit the size of local caches. To free up this space, use rm -rf .wireit/*/cache. Automatic cache size limits will be added in an upcoming release, tracked at wireit#71.

GitHub Actions caching

In GitHub Actions mode, Wireit caches output files to the GitHub Actions cache service. This service is available whenever running in GitHub Actions, and is free for all GitHub users.

ℹ️ GitHub Actions cache entries are automatically deleted after 7 days, or if total usage exceeds 10 GB (the least recently used cache entry is deleted first). See the GitHub Actions documentation for more details.

To enable caching on GitHub Actions, add the following uses clause to your workflow. It can appear anywhere before the first npm run or npm test command:

- uses: google/wireit@setup-github-actions-caching/v1

Example workflow

# File: .github/workflows/tests.yml

name: Tests
on: [push, pull_request]
jobs:
  tests:
    os: ubuntu-20.04
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16
          cache: true

      # Set up GitHub Actions caching for Wireit.
      - uses: google/wireit@setup-github-actions-caching/v1

      # Install npm dependencies.
      - run: npm ci

      # Run tests. Wireit will automatically use
      # the GitHub Actions cache whenever possible.
      - run: npm test

Cleaning output

Wireit can automatically delete output files from previous runs before executing a script. This is helpful for ensuring that every build is clean and free from outdated files created in previous runs from source files that have since been removed.

Cleaning is enabled by default as long as the output array is defined. To change this behavior, set the wireit.<script>.clean property to one of these values:

Setting	Description
true	Clean before every run (the default).
"if-file-deleted"	Clean only if an input file has been deleted since the last run. Use this option for tools that have incremental build support, but do not clean up outdated output when a source file has been deleted, such as tsc --build (see TypeScript for more on this example.)
false	Do not clean. Only use this option if you are certain that the script command itself already takes care of removing outdated files from previous runs.

Watch mode

In watch mode, Wireit monitors all files of a script, and all files of its transitive dependencies, and when there is a change, it re-runs only the affected scripts. To enable watch mode, ensure that the files array is defined, and add the --watch flag:

npm run <script> --watch

The benefit of Wireit's watch mode over built-in watch modes are:

• Wireit watches the entire dependency graph, so a single watch command replaces many built-in ones.
• It prevents problems that can occur when running many separate watch commands simultaneously, such as build steps being triggered before all preceding steps have finished.

Failures and errors

By default, when a script fails (meaning it returned with a non-zero exit code), all scripts that are already running are allowed to finish, but new scripts are not started.

In some situations a different behavior may be better suited. There are 2 additional modes, which you can set with the WIREIT_FAILURES environment variable. Note that Wireit always ultimately exits with a non-zero exit code if there was a failure, regardless of the mode.

Continue

When a failure occurs in continue mode, running scripts continue, and new scripts are started as long as the failure did not affect their dependencies. This mode is useful if you want a complete picture of which scripts are succeeding and which are failing.

WIREIT_FAILURES=continue

Kill

When a failure occurs in kill mode, running scripts are immediately killed, and new scripts are not started. This mode is useful if you want to be notified as soon as possible about any failures.

WIREIT_FAILURES=kill

Package locks

By default, Wireit automatically treats package-lock.json files in the package directory, plus all parent directories, as input files. This is useful because installing or upgrading your dependencies can affect the behavior of your scripts, so it's important to re-run them whenever your dependencies change.

If you are using an alternative package manager instead of npm, then your package lock files might be named something else. Some examples are:

• Yarn: yarn.lock (configurable)
• pnpm: pnpm-lock.yaml

To change the name of the package lock files Wireit should look for, specify it in the wireit.<script>.packageLocks array. Wireit will look for the given filenames in the script's directory, as well as in all of its parent directories. You can specify multiple filenames here, if needed.

{
  "scripts": {
    "build": "wireit"
  },
  "wireit": {
    "build": {
      "command": "tsc",
      "files": ["src/**/*.ts", "tsconfig.json"],
      "output": ["lib/**"],
      "packageLocks": ["yarn.lock"]
    }
  }
}

If you're sure that a script isn't affected by dependencies at all, you can turn off this behavior entirely to improve your cache hit rate by setting wireit.<script>.packageLocks to [].

Recipes

This section contains advice about integrating specific build tools with Wireit.

TypeScript

{
  "scripts": {
    "ts": "wireit"
  },
  "wireit": {
    "ts": {
      "command": "tsc --build --pretty",
      "clean": "if-file-deleted",
      "files": ["src/**/*.ts", "tsconfig.json"],
      "output": ["lib/**", ".tsbuildinfo"]
    }
  }
}

• Set "incremental": true and use --build to enable incremental compilation, which significantly improves performance.
• Include .tsbuildinfo in output so that it is reset on clean builds. Otherwise tsc will get out of sync and produce incorrect output.
• Set "clean": "if-file-deleted" so that you get fast incremental compilation when sources are changed/added, but also stale outputs are cleaned up when a source is deleted (tsc does not clean up stale outputs by itself).
• Include tsconfig.json in files so that changing your configuration re-runs tsc.
• Use --pretty to get colorful output despite not being attached to a TTY.

ESLint

{
  "scripts": {
    "lint": "wireit"
  },
  "wireit": {
    "lint": {
      "command": "eslint --color --cache --cache-location .eslintcache .",
      "files": ["src/**/*.ts", ".eslintignore", ".eslintrc.cjs"],
      "output": []
    }
  }
}

• Use --cache so that eslint only lints the files that were added or changed since the last run, which significantly improves performance.
• Use --color to get colorful output despite not being attached to a TTY.
• Include config and ignore files in files so that changing your configuration re-runs eslint.

Reference

Configuration

The following properties can be set inside wireit.<script> objects in package.json files:

Property	Type	Default	Description
command	string	undefined	The shell command to run.
dependencies	string[]	undefined	Scripts that must run before this one.
files	string[]	undefined	Input file glob patterns, used to determine the fingerprint.
output	string[]	undefined	Output file glob patterns, used for caching and cleaning.
clean	boolean | "if-file-deleted"	true	Delete output files before running.
packageLocks	string[]	['package-lock.json']	Names of package lock files.

Dependency syntax

The following syntaxes can be used in the wireit.<script>.dependencies array:

Example	Description
foo	Script named "foo" in the same package.
../foo:bar	Script named "bar" in the package found at ../foo (details).

Environment variables

The following environment variables affect the behavior of Wireit:

Variable	Description
WIREIT_FAILURES	How to handle script failures. Options: no-new (default): Allow running scripts to finish, but don't start new ones. continue: Allow running scripts to continue, and start new ones unless any of their dependencies failed. kill: Immediately kill running scripts, and don't start new ones.
WIREIT_PARALLEL	Maximum number of scripts to run at one time. Defaults to 4×CPUs. Must be a positive integer or infinity.
WIREIT_CACHE	Caching mode. Defaults to local unless CI is true, in which case defaults to none. Automatically set to github by the google/wireit@setup-github-actions-caching/v1 action. Options: local: Cache to local disk. github: Cache to GitHub Actions. none: Disable caching.
CI	Affects the default value of WIREIT_CACHE. Automatically set to true by GitHub Actions and most other CI (continuous integration) services. Must be exactly true. If unset or any other value, interpreted as false.

Glob patterns

The following glob syntaxes are supported in the files and output arrays:

Example	Description
foo	The file named foo, or if foo is a directory, all recursive children of foo.
foo/*.js	All files directly in the foo/ directory which end in .js.
foo/**/*.js	All files in the foo/ directory, and all recursive subdirectories, which end in .js.
foo.{html,js}	Files named foo.html or foo.js.
!foo	Exclude the file or directory foo from previous matches.

Also note these details:

• Paths should always use / (forward-slash) delimiters, even on Windows.
• Paths are interpreted relative to the current package even if there is a leading / (e.g. /foo is the same as foo).
• Whenever a directory is matched, all recursive children of that directory are included.
• files are allowed to reach outside of the current package using e.g. ../foo. output files cannot reference files outside of the current package.
• Symlinks in input files are followed, so that they are identified by their content.
• Symlinks in output files are cached as symlinks, so that restoring from cache doesn't create unnecessary copies.
• The order of !exclude patterns is significant.
• Hidden/dot files are matched by * and **.
• Patterns are case-sensitive (if supported by the filesystem).

Fingerprint

The following inputs determine the fingerprint for a script. This value is used to determine whether a script can be skipped for incremental build, and whether its output can be restored from cache.

• The command setting.
• The extra arguments set on the command-line.
• The clean setting.
• The output glob patterns.
• The SHA256 content hashes of all files matching files.
• The SHA256 content hashes of all files matching packageLocks in the current package and all parent directories.
• The system platform (e.g. linux, win32).
• The system CPU architecture (e.g. x64).
• The system Node version (e.g. 16.7.0).
• The fingerprint of all transitive dependencies.

When using GitHub Actions caching, the following input also affects the fingerprint:

• The ImageOS environment variable (e.g. ubuntu20, macos11).

Requirements

Wireit is supported on Linux, macOS, and Windows.

Wireit is supported on Node Current (18), Active LTS (16), and the most recent Maintenance LTS (14). See Node releases for the schedule.

Wireit is supported on the npm versions that ship with the above supported Node versions (5 and 6), Yarn Classic (1), and on the latest versions of Yarn Berry (3) and pnpm (7).

Related tools

Wireit shares a number of features with these other great tools, and we highly recommend you check them out too:

• Nx
• Turborepo
• Chomp
• Bazel

Here are some things you might especially like about Wireit:

• Feels like npm. When you use Wireit, you'll continue typing the same npm commands you already use, like npm run build and npm test. There are no new command-line tools to learn, and there's only one way to run each script. Your script config stays in your package.json, too. Wireit is designed to be the minimal addition to npm needed to get script dependencies and incremental build.

• Caching with GitHub Actions. Wireit supports caching build artifacts and test results directly through GitHub Actions, without any extra third-party services. Just add a single uses: line to your workflows.

• Watch any script. Want to automatically re-run your build and tests whenever you make a change? Type npm test --watch. Any script you've configured using Wireit can be watched by typing --watch after it.

• Great for single packages and monorepos. Wireit has no opinion about how your packages are arranged. It works great with single packages, because you can link together scripts within the same package. It also works great with any kind of monorepo, because you can link together scripts across different packages using relative paths.

• Complements npm workspaces. We think Wireit could be the missing tool that unlocks the potential for npm workspaces to become the best way to set up monorepos. To use Wireit with npm workspaces, you'll just use standard npm workspace commands like npm run build -ws.

• Adopt incrementally. Wireit scripts can depend on plain npm scripts, so they can be freely mixed. This means you can use Wireit only for the parts of your build that need it most, or you can try it out on a script-by-script basis without changing too much at the same time.

Contributing

See CONTRIBUTING.md