docker-library / bashbrew

Canonical build tool for the official images

Bashbrew (bashbrew)

$ bashbrew --help
NAME:
   bashbrew - canonical build tool for the official images

USAGE:
   bashbrew [global options] command [command options] [arguments...]

VERSION:
   v0.1.0

COMMANDS:
     list, ls    list repo:tag combinations for a given repo
     build       build (and tag) repo:tag combinations for a given repo
     tag         tag repo:tag into a namespace (especially for pushing)
     push        push namespace/repo:tag (see also "tag")
     put-shared  update shared tags in the registry (and multi-architecture tags)
     help, h     Shows a list of commands or help for one command
   plumbing:
     children, offspring, descendants, progeny  print the repos built FROM a given repo or repo:tag
     parents, ancestors, progenitors            print the repos this repo or repo:tag is FROM
     cat                                        print manifest contents for repo or repo:tag
     from                                       print FROM for repo:tag

GLOBAL OPTIONS:
   --debug                  enable more output (esp. all "docker build" output instead of only output on failure) [$BASHBREW_DEBUG]
   --no-sort                do not apply any sorting, even via --build-order
   --arch value             the current platform architecture (default: "amd64") [$BASHBREW_ARCH]
   --namespace value        a repo namespace to act upon/in [$BASHBREW_NAMESPACE]
   --constraint value       build constraints (see Constraints in Manifest2822Entry) [$BASHBREW_CONSTRAINTS]
   --exclusive-constraints  skip entries which do not have Constraints
   --arch-namespace value   architecture to push namespace mappings for creating indexes/manifest lists ("arch=namespace" ala "s390x=tianons390x") [$BASHBREW_ARCH_NAMESPACES]
   --config value           where default "flags" configuration can be overridden more persistently (default: "/home/tianon/.config/bashbrew") [$BASHBREW_CONFIG]
   --library value          where the bodies are buried (default: "/home/tianon/docker/official-images/library") [$BASHBREW_LIBRARY]
   --cache value            where the git wizardry is stashed (default: "/home/tianon/.cache/bashbrew") [$BASHBREW_CACHE]
   --help, -h, -?           show help
   --version, -v            print the version

Installing

Pre-built binaries are available to download from Jenkins (for all supported architectures).

(For building bashbrew yourself, it's a pretty standard Go application.)

Usage

Docker version 1.10 or above is required for use of Bashbrew.

In general, bashbrew build some-repo or bashbrew build ./some-file should be sufficient for using the tool at a surface level, especially for testing. For more complex usage, please see the built-in help (bashbrew --help, bashbrew build --help, etc).

Configuration

The default "flags" configuration is in ~/.config/bashbrew/flags, but the base path can be overridden with --config or BASHBREW_CONFIG (technically, the full path to the default flags configuration file is ${BASHBREW_CONFIG:-${XDG_CONFIG_HOME:-$HOME/.config}/bashbrew}/flags).

To set a default namespace for specific commands only:

Commands: tag, push
Namespace: officialstaging

To set a default namespace for all commands:

Namespace: jsmith

A more complex example:

# comments are allowed anywhere (and are ignored)
Library: /mnt/docker/official-images/library
Cache: /mnt/docker/bashbrew-cache
Constraints: aufs, docker-1.9, tianon
ExclusiveConstraints: true

# namespace just "tag" and "push" (not "build")
Commands: tag, push
Namespace: tianon

Commands: list
ApplyConstraints: true

Commands: tag
Debug: true

In this example, bashbrew tag will get both Namespace and Debug applied (options are additive).

bashbrew's Issues

Allow to override the repository name and config location

I did see on
That the image was built as docker and not phpmyadmin because of:

bashbrew/scripts/github-actions/generate.sh

Line 4 in b4880ac

image="${GITHUB_REPOSITORY##*/}" # "python", "golang", etc

So I did a small tweak to change this (I was thinking that that would make my config.sh work but it is not read)

  generate-jobs:
    name: Generate Jobs
    runs-on: ubuntu-latest
    outputs:
      strategy: ${{ steps.generate-jobs.outputs.strategy }}
    steps:
      - uses: actions/checkout@v3
      - id: generate-jobs
        name: Generate Jobs
        # See: https://github.com/docker-library/bashbrew/blob/b4880ac5f1bf6b9340536f1c5325afa9733bd619/scripts/github-actions/generate.sh#L4
        run: |
          git clone --depth 1 https://github.com/docker-library/bashbrew.git -b master ~/bashbrew
          export GITHUB_REPOSITORY="phpmyadmin"
          strategy="$(~/bashbrew/scripts/github-actions/generate.sh)"
          jq . <<<"$strategy" # sanity check / debugging aid
          echo "::set-output name=strategy::$strategy"

arm32v* (base) images missing variant, causing errors on Docker 20.10+

See alpinelinux/docker-alpine#194 for an example -- docker pull --platform linux/arm/v6 arm32v6/alpine:edge is fine, but docker run -it --rm --platform linux/arm/v6 arm32v6/alpine:edge errors with "docker: Error response from daemon: image with reference arm32v6/alpine:edge was found but does not match the specified platform: wanted linux/arm/v6, actual: linux/arm/v7." (because the image lacks variant so the containerd platforms code "helpfully" assumes for us that it must be v7).

"bashbrew push" does not update repo when tag is moved

When a tag is moved to refer to another image, the docker repo will not be updated to reflect this when pushing images with bashbrew. The only thing that is checked is the updated time, so if the image that the tag is moved to was built before the image the tag referred to previously, it won't be updated in the repo.

Example:
Given that image tagged v2 was built before image tagged v1:

Tags: v1, test
GitCommit: <SHA1>

Tags: v2
GitCommit: <SHA1>

moving test tag to image with v2 tag

Tags: v1
GitCommit: <SHA1>

Tags: v2, test
GitCommit: <SHA1>

bashbrew: Generic support for Docker Registries

Right now the tool depends on the DockerHub API which isn't compatible with the Docker Registry HTTP API V2.

This makes bashbrew less portable and not suited for managing docker libraries running on other registries.

Would you be open to supporting patches that:

1. Provide config flag(s) to instruct bashbrew to use the official registry API for tag metadata scraping
2. Add the necessary authentication support to use the registry API

Import OCI (base) images as-is

In the case of base images (debian, alpine, ubuntu, etc), using a Dockerfile as our method of ingestion doesn't really buy us very much. It made sense at the time it was implemented ("all Dockerfile, all the time"), but at this point they're all some variation on FROM scratch \n ADD foo.tar.xz / \n CMD ["/bin/some-shell"], and cannot reasonably be "rebuilt" when their base image changes (which is one of the key functions of the official images) since they are the base images in question.

Functionally, consuming a tarball in this way isn't that much different from consuming a raw tarball that's part of, say, an OCI image layout (https://github.com/opencontainers/image-spec/blob/v1.0.2/image-layout.md) -- it's some tarball plus some metadata about what to do with it.

For less trivial images, there's a significant difference (and I'm not proposing to use this for anything beyond simple one-layer base images), but for a single layer this would be basically identical.

As a more specific use case, the Debian rootfs.tar.xz files are currently 100% reproducible. Unfortunately, some of that gets lost when it gets imported into Docker, and thus it takes some additional effort to get from the Docker-generated rootfs back to the original debuerreotype-generated file (see debuerreotype/docker-debian-artifacts#147 (comment) for an example where I've done so).

With the ability to consume an OCI image directly, I would be able to use something like debuerreotype/debuerreotype#108 to go even further and have a 100% fully reproducible image digest as well, and it would be easier to trace a given published image back to the reproducible source generated by the upstream tooling.

Git-LFS support

Hello,

is it possible to use lfs-enabled git repositories in the official images' specs? according to that, git-lfs binary must be installed on the side they're pulled/cloned to.

Thanks in advance.

Consider making pgp-happy-eyeballs optional

Perhaps autodetection or just a manual opt-out:

bashbrew/scripts/github-actions/generate.sh

Lines 167 to 172 in 9df9222

	(
	"# PGP Happy Eyeballs",
	"git clone --depth 1 https://github.com/tianon/pgp-happy-eyeballs.git ~/phe",
	"~/phe/hack-my-builds.sh",
	"rm -rf ~/phe"
	)

docker-library/python#599 (comment)

Official image of bashbrew (or the build-tools for the official images)

Is there any official image of bashbrew or in general of the build-tools for the official images? I see the [https://github.com/docker-library/official-images/blob/master/Dockerfile](Dockerfile here) but cannot find a way to pull the image from the official library? Any plans on publishing this if it is not already the case.
Thanks.

question: convincing generate.sh to do a buildkit build

I'm trying to work out how to make generate.sh output a buildkit build on this criteria. Where is .builder = "buildkit" set?

bashbrew/scripts/github-actions/generate.sh

Line 81 in 10343d3

elif .builder == "buildkit" then

This is so I can access the COPY --chmod synax (ref) and can avoid another layer like:

https://github.com/MariaDB/mariadb-docker/pull/593/files#diff-06aaf016508acc42e173e462fdda2de06e46f80227174d6bcaf20019657a5efeR31-R33

Consider using sprig

We should spend more time looking at https://masterminds.github.io/sprig/ and determine whether/how much of https://github.com/docker-library/bashbrew/blob/8e42901deafd7baaf313ad4b655f85507b001746/pkg/templatelib/lib.go we could get rid of by using that instead (since it implements many of the same functions like first, last, ternary, etc - some just with a different name like toJson vs our json).

bashbrew: use with local repository

I know it's not one of the intended features, but i'm looking to reuse bashbrew for personnal repositories. Currently, it's only possible with bashbrew to reference git repository as reference for an image.

Having a verb in the rfc2822 config file for targeting a local directory could be useful in my case as I would like to have the library-file stored in the individual project rather than in a central repository.

Something like "LocalDirectory"

First, does this feature exists ? Maybe I did not look correctly.
Second, if not, would it fit the spirit of bashbrew to add such feature ?
Is bashbrew really tied to a git repository for the tagging process ?

Deprecate single line library file format

This issue is to declare and track the intent to drop the single line format. This is mostly to make parsing errors clear (rather than the current of printing both errors and having the user guess which is the issue).

We don't have releases often, so it would make most sense to pick some dates at which it would begin printing warnings on the single line format and then a date at which it will fail to parse them. Rough timeline would be to start warning sometime in September and then drop support at the end of the year completely remove the single line format before November.

Alternatively we could skip the warning period and just make a new release with single line parsing dropped since bashbrew is still pre 1.0. We would need to migrate any library files in official-images before the release.

(RFC 2822 manifest file format was added in June 2016: docker-library/official-images#1793)

• deprecate by adding warnings to bashbrew skipping
  • make a release
• remove single line parsing
  • make a release (maybe a major bump like 1.0)
• convert any current library/ files in official-images to RFC 2822 format (grep -E ': (http|git).*@' library/*)
  • arangodb
  • clearlinux
  • couchbase
  • sourcemage
  • thrift