paketo-buildpacks / jam

jam

jam is a command-line tool for buildpack authors and users. The jam name is simply a play on the idea of "packaging" or "packing" a buildpack.

jam comes with the following commands:

• create-stack : create a CNB stack
• help : help about any command
• pack : package buildpack
• summarize : summarize buildpackage
• update-builder : update builder
• update-buildpack : update buildpack
• update-dependencies : update all depdendencies in a buildpack.toml according to metadata.constraints

The jam executable can be installed by downloading the latest version from the Releases page. Once downloaded, buildpacks can be created from a source repository using the pack command like this:

jam pack \
  --buildpack ./buildpack.toml \
  --stack io.paketo.stacks.tiny \
  --version 1.2.3 \
  --offline \
  --output ./buildpack.tgz

Building stack images on linux

In order to build stack images on linux, you will need to first install packages to enable emulation of the arm64 instruction set. This is not an issue on other operating systems (e.g. OSX) because Docker already runs in a Virtual Machine with emulation support enabled.

For example, to enable emulation on Ubuntu 2204 (Jammy), run the following command:

sudo apt-get install qemu binfmt-support qemu-user-static

jam's Issues

Markdown summary has strange output for `*` stack

When summarizing a buildpack in markdown format, the result can be something like the following, when a * stack is included:

A couple of options:

• Wrap entries in a ` so that they are rendered literally (looks better for stack IDs anyway), OR
• Replace the entire list with - Any stack

Or even both, as the first option is a nice improvement anyway

Version: This behavior exists on the current version of jam as of this writing (v1.1.1).

Failure: Asset Validation Workflow

Release asset validation workflow failed.

Tests rely on brittle umask behavior

Expected Behavior

Tests pass out of the box irrespective of umask value.

Current Behavior

I cloned jam and ran the tests, and I observed that some tests fail. Specifically, there are some tests (e.g. this one) that assume the local umask is set to 0022. My default umask was set to 0002, so when I cloned the repo the files were created with permissions: 0775 instead of the assumed 0755.

Possible Solution

There are two possible ways to fix this, depending on whether jam should preserve the original permissions of the files being bundled:

• Update the tests to verify the bundled permissions are the same as the original file permissions. Do this if we don't particularly care what the file permissions of the bundled files are.

OR

• Explicitly write the desired permissions when bundling the file (similar to this). Do this if we want to be sure that the bundled files have known permissions.

Steps to Reproduce

Run the following commands

1. umask = 0022
2. git clone https://github.com/paketo-buildpacks/jam /tmp/jam
3. cd /tmp/jam
4. go test -v ./internal/...

Observe test failures e.g.:

=== RUN   TestUnitInternal/jam/internal/FileBundler/Bundle/returns_a_list_of_cargo_files
    file_bundler_test.go:53:
        Expected
            <fs.FileMode>: 509
        to equal
            <fs.FileMode>: 493

Motivations

I am trying to run the jam tests locally and it is inconvenient to have to temporarily change my umask before cloning (or manually update the permissions of all relevant files after cloning).

`jam create-stack`: Produce SBOMs for stacks in a standard SBOM format

While working on paketo-buildpacks/jammy-tiny-stack#2, it was brought up that the current mechanism for generating package receipts (using dpkg -l) is brittle; it won't work for non-Ubuntu stacks. It'd be better for Paketo's stacks automation – and probably for stack consumers – if the list of packages installed in the stack were generated in a standard SBOM format (e.g. CycloneDX) and returned as an output of jam create-stack.

`jam update-builder` is clobbering extensions

This can be seen in the this PR paketo-community/builder-ubi-base#6 instead of updating the extensions the section is clobbered as it is not currently accounted for in the update logic. The update-builder command should be updated to account for extension and to attempt to upgrade them.

create-stack: stream output from Docker

Currently when running jam create-stack it appears to hang while building the images:

Building io.buildpacks.stacks.jammy.full
  Building on linux/amd64
    Building base images

In the case of the full stack, this process can take many minutes.

When running a docker build command, there is constant streaming output.

It would be a significant UX improvement to jam create-stack if we could stream this same output (or similar) while we build the stack.

Create stack without docker daemon

Currently using jam create-stack requires access to a docker daemon. That put some burden on the CI since this is not always easy, e.g. running on kubernetes.

Describe the Enhancement

Remove all direct dependencies to the docker daemon.

Possible Solution

For example using a different kind of sink for the go-containerregistry. Using BuildKit should still be fine since it can be used without the docker daemon.

Motivation

It would be easier to use jam create-stack, especially in a CI system that do not have easy access to a docker daemon.

Please provide `arm64` versions of the build artifacts in future GitHub releases

Please provide arm64 versions of the build artifacts in future GitHub releases

Describe the Enhancement

Currently the automated build/release mechanism only builds/publishes artifacts for amd64 architecture. With the proliferation of arm64 processors (Apple M1/M2, Raspberry PI, AWS Graviton, etc) providing arm64 artifacts in the releases would benefit those using these processors.

Possible Solution

golang builds can easily output binaries for arm64 architecture in addition to the existing amd64 architecture. No source code needs to change, just the build automation scripts. I can submit a pull-request to do this.

Motivation

Want to take advantage of exciting new ARM64 based hardware and pre-built ARM64 jam binaries would be essential in creating stack and builder images for that platform.

create-stack: update /etc/os-release

Currently, when using jam create-stack we have to manually update the values in /etc/os-release (e.g. tiny stack and we do something similar for the base and full stacks).

This file could be generated by jam create-stack rather than requiring users to put this in their Dockerfile.

/etc/os-release is present on all linux operating systems, so we could write it only when the platform contains linux.

jam --version doesn't match jam Github Release version

What happened?

• What were you attempting to do?
  I wanted to download the latest version of jam, then check the version I'd downloaded with jam --version. I downloaded the jam-darwin artifact attached to the v1.1.0 release.

• What did you expect to happen?
  I expected the printed version to match the version on the release I'd downloaded.

• What was the actual behavior? Please provide log output, if possible.

jam version
jam 1.0.3

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

Add verbose output to create-stack command

I can successfully create a stack with create-stack but there are times where I would like to show all of the output that I would get if there is an error (e.g. the dockerfile steps).

I'd like this flag to take the form -v / --verbose.

I would suggest that this flag operates on all commands, not just create-stack, but create-stack is the command I care most about at this time.

Error: No such file or directory for docker >=25.0.0

Expected Behavior

jam create-stack should create a build.oci and run.oci output files

Current Behavior

jam create-stack fails with bellow error :

Using jam 2.7.0
Building io.buildpacks.stacks.jammy
  Building on linux/amd64
    Building base images
Error: open /tmp/2573542004/wsnpmg4jaz/blobs/sha256/d101c9453715a978a2a520f553588e77dfb4236762175eba61c5c264a449c75d: no such file or directory
Usage:
  jam create-stack [flags]

Flags:
      --build-output string   path to output the build image OCI archive (required)
      --build-ref string      reference that specifies where to publish the build image (required)
      --config string         path to a stack descriptor file (required)
  -h, --help                  help for create-stack
      --label strings         additional image label to be added to build and run image
      --publish               publish to a registry
      --run-output string     path to output the run image OCI archive (required)
      --run-ref string        reference that specifies where to publish the run image (required)
      --secret strings        secret to be passed to your Dockerfile
      --unbuffered            do not buffer image contents into memory for fast access

failed to execute: open /tmp/2573542004/wsnpmg4jaz/blobs/sha256/d101c9453715a978a2a520f553588e77dfb42367

Possible Solution

Steps to Reproduce

1. Install Docker version 25 and above
2. git clone https://github.com/paketo-buildpacks/jammy-base-stack.git
3. cd jammy-base-stack
4. ./scripts/create.sh

Motivations

Add semantic version label workflows

In our other repositories, like https://github.com/paketo-buildpacks/packit and our buildpacks, we have moved to using labeling workflows to auto-label and check for labels on PRs. This also feeds into the the draft release workflows to make version resolution simpler.

We should move jam automation over to this model for consistency with our other repositories. It also makes life easier when we have to cut releases that aren't patches.

Implement Dependency Management RFC 0005: New `jam update-dependencies`

Per Dependency Management RFC Phase 2, part of the new dependency update workflow there is an "assembly" step in which new dependency versions and metadata get added to the buildpack.toml.

This issue is to create a new jam command, which will eventually replace the existent jam update-dependencies command to update the buildpack.toml. The command will be extremely similar in that it will update the buildpack.toml, but it will no longer query the dep-server for metadata, and will instead receive a metadata.json.
A key difference is that it will expect that the metadata provided has already been checked against existent buildpack.toml versions and the version constraints. This means, the main job of the command is to take all of the new versions to be added, and appropriately sub them into the buildpack.toml to satisfy constraints (like number of patches for a version line).

Check out the RFC for details about what the command should do, and how it should fit into the future dependency update workflows.

Note that the new command cannot immediately replace jam update-dependencies because the old command will be needed for a period of time until migration to the new dependency management strategy is complete. The new command will need to be given a different name for the time being in order to avoid breakages in our current process.

Acceptance Criteria

I package up the version of jam on the related PR and:

• The jam update-dependencies --buildpack-file <path to buildpack.toml> command still works the same way as it does now, it will use the dep-server to update the buildpack.toml.

• The jam update-dependencies command can also take in a metadata-file argument, which must be the path to a JSON file containing metadata as outlined in the RFC.

  • The metadata may also now contain duplicate versions, which differ by the set of stacks. In this case the jam update-dependencies command adds both entries to the buildpack.toml. See this metadata file as an example, which can be used alongside this example buildpack.toml file.

Failure to build stack when build and run Dockerfiles are the same

In paketo-buildpacks/dotnet-core#653, a user (@lamw) reported seeing a failure when building a Powershell-based image on Mac:

❯ jam create-stack --config stack.toml --build-output build.oci --run-output run.oci
Building photon-powershell
  Building on linux/amd64
    Building base images
      Build complete for base images
    build: Decorating base image
      Adding CNB_* environment variables
      Adding io.buildpacks.stack.* labels
      Creating cnb user
    run: Decorating base image
      Adding io.buildpacks.stack.* labels
      Creating cnb user
    build: Updating image
    run: Updating image
Error: Error: No such image: paketo.io/stack/87zgdlj0a5
Usage:
  jam create-stack [flags]

Flags:
      --build-output string   path to output the build image OCI archive (required)
      --config string         path to a stack descriptor file (required)
  -h, --help                  help for create-stack
      --run-output string     path to output the run image OCI archive (required)
      --secret strings        secret to be passed to your Dockerfile

failed to execute: Error: No such image: paketo.io/stack/87zgdlj0a5

I was able to reproduce this error on my Mac, when passing in the same stack.toml file, and setting up a powershell Dockerfile .

It appears that the run image build phase succeeds and the image is on the Docker daemon during the build, but for some reason the run image is missing from the Docker daemon when the run image update step runs.

It looks like this error does not occur when building the Tiny stack, for example on Mac

Issue

We should investigate why this is occurring and put in a fix to the create-stack codebase. If it's a problem with the Dockerfile used, we should make usage information clearer for users.
We should also try to reproduce this failure on a linux machine and see if the same error crops up.

Failure: Create Draft Release workflow

Create Draft Release workflow failed.

Support dependency filters when packing a buildpack offline

Describe the Enhancement

Currently when you package a buildpack offline with Jam, it will include all dependencies. With libpak you can use --dependency-filter multiple times to provide regular expressions that match dependencies to include.

Additionally you can use --strict-filters so that the regular expression needs to match the dependency ID as well as the Version. This allows users to provide a filter of (jre|^17.*) to only include that dependency for a specific version.

Possible Solution

The same functionality as libpak create-package. See above.

Motivation

Packaging offline buildpacks will often include many more dependencies than you know you will need and creates huge images that need to be pulled at build time.

Remove all references to packit.Cargo

Going through some of the tests, we reference cargo/jam/internal in many places. We should go through all of our tests and files and make sure we are referring to paketo-buildpacks/jam/internal everywhere instead of packit/cargo

create-stack: add error wrapping

When trying to build stacks on my local workstation, I ran into a few failures. Some were Docker related, some were system related. I think one case, the docker build step failed (corresponding to this line of code), but the lack of error returned here made the error challenging to pinpoint until I went through the code and debugged.

It would be great if we could return a bit more information about errors we return, especially at key points in the stack-creation process such as the image-building phase.

Get a default stack descriptor file for `create-stack`

With #52, the jam create-stack command was introduced. This command is very useful for simplified stack creation, but requires users to provide a stack descriptor file. This file is defined clearly, but still requires a user to go and look at this outline, and create their own.

Issue

We should have a way to provide users with a basic stack descriptor file, likely through a jam create-stack-descriptor or by providing a default file if none are provided to the create-stack command.

Outcome

Some mechanism is available in jam to easily get a default stack descriptor file without having to create one from scratch.

Implement Stacks RFC0003: Stack Descriptor and Tooling

RFC

Summary

The stack-building codebase, create-stack should be moved into the jam tooling CLI, and allow users to configure their stack build via a common configuration file.

Implementation

The create-stack command will be refactored to support this new configuration file format and the features it allows. The codebase will then be moved into the jam codebase as a command within that tool. This aligns more with how the project maintains its tooling and allows us to keep tooling consolidated under the tooling subteam.

The stacks team will be granted CODEOWNERS status for this addition to the codebase.

March 3: Remove `jam update-dependencies` from API

In #195 we issued a warning that we will be removing the ability to use the jam update-dependencies command with the dep-server or another API in favour of using a metadata file. Once the dep-server is shut down on March 3, 2022, we should remove the associated jam behaviour at the same time.

Add jam version command

What happened?

• What were you attempting to do?
  Find out which version of the jam CLI I had installed on my workstation

• What did you expect to happen?
  jam version and/or jam --version and/or jam -v would return the semver version of the installed CLI.

• What was the actual behavior? Please provide log output, if possible.

jam --version
Error: unknown flag: --version
Usage:
  jam [command]

Available Commands:
  completion          generate the autocompletion script for the specified shell
  help                Help about any command
  pack                package buildpack
  summarize           summarize buildpackage
  update-builder      update builder
  update-buildpack    update buildpack
  update-dependencies updates all depdendencies in a buildpack.toml according to metadata.constraints

Flags:
  -h, --help   help for jam

Use "jam [command] --help" for more information about a command

Checklist

• I have included log output.
• The log output includes an error message.
• I have included steps for reproduction.

Implement Buildpack RFC 0070: New Buildpack Descriptor Keys

Summary

Buildpack RFC0070 introduces some new fields into the buildpack.toml file. These fields include description, keywords, and licenses. The specific changes to the buildpack.toml would look like the following:

[buildpack]
description = "<buildpack description>"
keywords = [ "<string>" ]

[[buildpack.licenses]]
type = "<string>"
uri = "<uri>"

Proposal

When we jam pack a buildpack that includes these new keys, and is specifying buildpack API >= 0.6, include these keys in the built buildpack artifact.

Document additional steps required to build arm64 stack on amd64 machine

Expected Behavior

It should be easy to discover how to build an linux/arm64 stack on a linux/amd64 machine.

I think we should add some steps to the readme for linux users. Specifically we should document that linux users need to install qemu emulation packages.

For ubuntu this looks like:

sudo apt-get install qemu binfmt-support qemu-user-static 

Current Behavior / Steps to Reproduce

I clone the repo and build jam and try to run the example stack as follows:

❯ ./build/jam-linux create-stack \
  --config ~/workspace/paketo-buildpacks/jam/integration/testdata/example-stack/stack.toml \
  --build-output /tmp/build.oci \
  --run-output /tmp/run.oci \
  --secret some-secret=my-secret-value

I see that the linux/amd64 stack is created successfully:

Building io.paketo.stacks.example
  Building on linux/amd64
    Building base images
      Build complete for base images
    build: Decorating base image
      Adding CNB_* environment variables
      Adding io.buildpacks.stack.* labels
      Adding io.buildpacks.stack.mixins label
      Adding io.paketo.stack.packages label
      Creating cnb user
    run: Decorating base image
      Adding io.buildpacks.stack.* labels
      Adding io.buildpacks.stack.mixins label
      Adding io.paketo.stack.packages label
      Creating cnb user
    build: Updating image
    run: Updating image

But I immediately see failures when trying to build the linux/arm64 stack:

  Building on linux/arm64
    Building base images
Error: build failed:
#1 [internal] load remote build context
#1 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 DONE 0.2s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5
#4 CACHED

#5 [internal] load metadata for docker.io/library/alpine:3.15.4
#5 DONE 0.0s

#6 [1/2] FROM docker.io/library/alpine:3.15.4@sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
#6 CACHED

#7 [2/3] RUN apk add --no-cache curl git jq openssl

executor failed running [/bin/sh -c apk add --no-cache ${packages}]: exit code: 1
Usage:
  jam create-stack [flags]

Flags:
      --build-output string   path to output the build image OCI archive (required)
      --config string         path to a stack descriptor file (required)
  -h, --help                  help for create-stack
      --run-output string     path to output the run image OCI archive (required)
      --secret strings        secret to be passed to your Dockerfile
      --unbuffered            do not buffer image contents into memory for fast access

failed to execute: build failed:
#1 [internal] load remote build context
#1 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 DONE 0.2s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:600e5c62eedff338b3f7a0850beb7c05866e0ef27b2d2e8c02aa468e78496ff5
#4 CACHED

#5 [internal] load metadata for docker.io/library/alpine:3.15.4
#5 DONE 0.0s

#6 [1/2] FROM docker.io/library/alpine:3.15.4@sha256:4edbd2beb5f78b1014028f4fbb99f3237d9561100b6881aabbf5acce2c4f9454
#6 CACHED

#7 [2/3] RUN apk add --no-cache curl git jq openssl

executor failed running [/bin/sh -c apk add --no-cache ${packages}]: exit code: 1

Possible Solution

I am able to run this same command with the same stack.toml on my macbook, and I notice that we run the tests in privileged mode in github actions. So I suspected it might be something to do with my local docker permissions.

A quick search led me to install the following emulation packages:

sudo apt-get install qemu binfmt-support qemu-user-static 

Once these were installed the stack creation worked.

Motivations

I would like to be able to use jam (and run its integration tests) on my linux machine easily, following well-documented instructions to setup any prerequisites.