pombredanne / cachi2

Cachi2

Cachi2 is a CLI tool that pre-fetches your project's dependencies to aid in making your build process hermetic.

To see if we support your package manager(s), please check the package managers section.

The primary intended use of Cachi2's outputs is for network-isolated container builds (see usage).

Table of contents

• Goals
• Installation
• Basic usage
• Development
• Package managers
• Project status

Goals

Please note that Cachi2 is rather picky, aiming to:

• encourage or enforce best practices
• enforce building from source - no pre-built artifacts, such as Python wheels
• never execute arbitrary code - looking at you setup.py (discouraged)
• keep the implementation simple

To play nicely with Cachi2, the build process for your project must be

• Defined - Cachi2 only fetches dependencies that are explicitly declared - typically in a lockfile generated by your package manager.

• Reproducible - Cachi2 will refuse to fetch a dependency if it's not pinned to an exact version. This goes for transitive dependencies as well (and ties to the Defined point). Most package managers pin all dependencies automatically in lockfiles.

• Secure - Even with a lockfile, your build is not truly safe from supply chain attacks (such as dependency confusion) unless you verify the checksums of all dependencies. If your package manager supports specifying the expected checksums, we strongly encourage you to make use of them.

  ⚠ Cachi2 will verify checksums if present, but doesn't require them by default. This may change in the future.

In return, Cachi2 will help make your build

• Auditable - by generating a manifest of all the dependencies that go into your build.

The ability to achieve the goals depends on the hermeticity of the build process. Ideally, you should try to isolate the build from both the internet and the underlying host system to avoid implicit dependencies, irreproducible behavior and whole hosts of other issues. Cachi2 itself is not a hermetic build system. We suggest you take advantage of existing technologies - such as containers - to achieve isolation (see usage).

Installation

Standalone

We do not distribute Cachi2 as a standalone package as of now.

To install Cachi2 for local development, see the development section.

Container image

quay.io/containerbuildsystem/cachi2:latest

The container is re-built automatically on every merge to the main branch.

You may wish to set up an alias to make local usage more convenient:

alias cachi2='podman run --rm -ti -v "$PWD:$PWD:z" -w "$PWD" quay.io/containerbuildsystem/cachi2:latest'

Note that the alias mounts the current working directory - the container will have access to files in that directory and nowhere else.

Basic usage

cachi2 fetch-deps \
  --source ./my-repo \
  --output ./cachi2-output \
  --package gomod

The fetch-deps command fetches your project's dependencies and stores them on your disk. You can then use these outputs to, say, build a container image.

See docs/usage.md for a more detailed, practical (cough) example of Cachi2 usage.

You might also like to check out cachi2 --help and the --help texts of the available subcommands.

Development

Virtualenv

Set up a virtual environment that has everything you will need for development:

make venv
source venv/bin/activate

This installs the Cachi2 CLI in editable mode, which means changes to the source code will reflect in the behavior of the CLI without the need for reinstalling.

You may need to install the following dependencies before creating the virtualenv:

dnf install python3.9 python3-virtualenv

The CLI also depends on the following non-Python dependencies:

dnf install golang-bin git

You should now have everything needed to try out the CLI or hack on the code in vim your favorite editor.

Coding standards

Cachi2's codebase conforms to standards enforced by a collection of formatters, linters and other code checkers:

• black (with a line-length of 100) for consistent formatting
• isort to keep imports sorted
• flake8 to (de-)lint the code and politely ask for docstrings
• mypy for type-checking. Please include type annotations for new code.
• pytest to run unit tests and report coverage stats. Please aim for (near) full coverage of new code.

Options for all the tools are configured in pyproject.toml and tox.ini.

Run all the checks that your pull request will be subjected to:

make test

Error message guidelines

We try to keep error messages friendly and actionable.

• If there is a known solution, the error message should politely suggest the solution.
  • Include a link to the documentation when suitable.
• If there is no known solution, suggest where to look for help.
• If retrying is a possible solution, suggest retrying and where to look for help if the issue persists.

The error classes aim to encourage these guidelines. See the errors.py module.

Running unit tests

Run all unit tests (but no other checks):

make test-unit

For finer control over which tests get executed, e.g. to run all tests in a specific file, activate the virtualenv and run:

tox -e python3.9 -- tests/unit/test_cli.py

Even better, run it stepwise (exit on first failure, re-start from the failed test next time):

tox -e python3.9 -- tests/unit/test_cli.py --stepwise

You can also run a single test class or a single test method:

tox -e python3.9 -- tests/unit/test_cli.py::TestGenerateEnv
tox -e python3.9 -- tests/unit/test_cli.py::TestGenerateEnv::test_invalid_format
tox -e python3.9 -- tests/unit/extras/test_envfile.py::test_cannot_determine_format

In short, tox passes all arguments to the right of -- directly to pytest.

Running integration tests

Build Cachi2 image and run all integration tests (but no other checks):

make test-integration

To run integration-tests with custom image, specify the CACHI2_IMAGE environment variable. Examples:

CACHI2_IMAGE=quay.io/containerbuildsystem/cachi2:{tag} tox -e integration
CACHI2_IMAGE=localhost/cachi2-${USER}:latest tox -e integration

Similarly to unit tests, for finer control over which tests get executed, e.g. to run only 1 specific test case, execute:

CACHI2_IMAGE=localhost/cachi2-${USER}:latest tox -e integration -- tests/integration/test_package_managers.py::test_packages[gomod_without_deps]

Package managers

Supported:

• gomod

Planned:

• pip (coming soon)
• npm
• yarn
• rubygems

Based on the supported package managers in the original Cachito.

gomod

Current version: 1.18 ^{1 2}

The gomod package manager works by parsing the go.mod file present in the source repository to determine which dependencies to download. Cachi2 does not parse this file on its own - rather, we rely on the go command to download and list the required dependencies.

From go 1.17 onward, the go.mod file includes all the transitively required dependencies of your application - see the section about Pruned module graphs in the 1.17 changelog. In previous go versions, the go.mod file included only direct dependencies. Cachi2 does support downloading and listing all transitive dependencies for earlier versions thanks to Go's backwards compatibility². Note that using go >= 1.17 in your project has the added benefit of downloading fewer dependencies (as noted in the changelog), in some cases drastically so.

See docs/gomod.md for more details.

Project status

Cachi2 was derived (but is not a direct fork) from Cachito and is still in early development phase.

Footnotes

1. Cachi2 expects to use a specific version of the go command when downloading dependencies. This is the version installed in the cachi2 container. We do not guarantee correctness if you run Cachi2 locally (outside the container) with a different Go version. You are free to use a different version to build your project. ↩

2. The go command promises to be backwards compatible with previous versions. If your go.mod file specifies the intended go version, Cachi2 should handle it appropriately. If your go version is higher than what Cachi2 uses, there is a good chance it will be compatible regardless, as long as the dependency resolution did not change between the two versions. For example, dependency resolution did change in go 1.18 but not in go 1.19. ↩ ↩²