GithubHelp home page GithubHelp logo

melange's Introduction

melange

Build apk packages using declarative pipelines.

Commonly used to provide custom packages for container images built with apko. The majority of apks are built for use with either the Wolfi or Alpine Linux ecosystems.

Key features:

  • Pipeline-oriented builds. Every step of the build pipeline is defined and controlled by you, unlike traditional package managers which have distinct phases.
  • Multi-architecture by default. QEMU is used to emulate various architectures, avoiding the need for cross-compilation steps.

Why

Secure software factories are the evolution of DevOps, allowing a user to prove the provenance of all artifacts incorporated into a software appliance. By building and capturing software artifacts into packages, DevOps teams can manage their software artifacts as if they were any other component of an image.

This is especially useful when building software appliances in the form of OCI container images with apko.

Installation

You can install Melange from Homebrew:

brew install melange

You can also install Melange from source:

go install chainguard.dev/melange@latest

You can also use the Melange container image:

docker run cgr.dev/chainguard/melange version

To use the examples, you'll generally want to mount your current directory into the container and provide elevated privileges, e.g.:

docker run --privileged -v "$PWD":/work cgr.dev/chainguard/melange build examples/gnu-hello.yaml

Running outside of a container requires Docker, but should also work with other runtimes such as podman.

Alternatively, if you're on a Mac, you can use the melange instructions for Lima to run an Alpine Linux VM.

Quickstart

A melange build file looks like:

package:
  name: hello
  version: 2.12
  epoch: 0
  description: "the GNU hello world program"
  copyright:
    - paths:
      - "*"
      attestation: |
        Copyright 1992, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2005,
        2006, 2007, 2008, 2010, 2011, 2013, 2014, 2022 Free Software Foundation,
        Inc.
      license: GPL-3.0-or-later
  dependencies:
    runtime:

environment:
  contents:
    repositories:
      - https://dl-cdn.alpinelinux.org/alpine/edge/main
    packages:
      - alpine-baselayout-data
      - busybox
      - build-base
      - scanelf
      - ssl_client
      - ca-certificates-bundle

pipeline:
  - uses: fetch
    with:
      uri: https://ftp.gnu.org/gnu/hello/hello-${{package.version}}.tar.gz
      expected-sha256: cf04af86dc085268c5f4470fbae49b18afbc221b78096aab842d934a76bad0ab
  - uses: autoconf/configure
  - uses: autoconf/make
  - uses: autoconf/make-install
  - uses: strip

We can build this with:

melange build examples/gnu-hello.yaml

or, with Docker:

docker run --privileged --rm -v "${PWD}":/work \
  cgr.dev/chainguard/melange build examples/gnu-hello.yaml

This will create a packages folder, with an entry for each architecture supported by the package. If you only want to build for the current architecture, you can add --arch $(uname -m) to the build command. Inside the architecture directory you should find apk files for each package built in the pipeline.

If you want to sign your apks, create a signing key with the melange keygen command:

melange keygen
 generating keypair with a 4096 bit prime, please wait...
 wrote private key to melange.rsa
 wrote public key to melange.rsa.pub

And then pass the --signing-key argument to melange build.

Debugging melange Builds

To include debug-level information on melange builds, edit your melange.yaml file and include set -x in your pipeline. You can add this flag at any point of your pipeline commands to further debug a specific section of your build.

...
pipeline:
  - name: Build Minicli application
    runs: |
      set -x
      APP_HOME="${{targets.destdir}}/usr/share/hello-minicli"
...

Default Substitutions

Melange provides the following default substitutions which can be referenced in the build file pipeline:

Substitution Description
${{package.name}} Package name
${{package.version}} Package version
${{package.epoch}} Package epoch
${{package.full-version}} ${{package.version}}-r${{package-epoch}}
${{targets.destdir}} Directory where targets will be stored
${{targets.subpkgdir}} Directory where subpackage targets will be stored

An example build file pipeline with substitutions:

pipeline:
  - name: 'Create tmp dir'
    runs: mkdir ${{targets.destdir}}/var/lib/${{package.name}}/tmp

More detailed documentation

Usage with apko

To use a melange built apk in apko, either upload it to a package repository or use a "local" repository. Using a local repository allows a melange build and apko build to run in the same directory (or GitHub repo) without using external storage. An example of this approach can be seen in the nginx-image-demo repo.

melange's People

Contributors

ajayk avatar alexsjones avatar amouat avatar bureado avatar cpanato avatar deitch avatar dependabot[bot] avatar developer-guy avatar dlorenc avatar elizafox avatar epsilon-phase avatar found-it avatar imjasonh avatar jdolitsky avatar jonjohnsonjr avatar joshrwolf avatar k4leung4 avatar kaniini avatar luhring avatar mattmoor avatar mesaglio avatar patflynn avatar priyawadhwa avatar puerco avatar rawlingsj avatar stormqueen1990 avatar strongjz avatar tuananh avatar vaikas avatar wlynch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

melange's Issues

Add "melange index" to simplify generation of APK index

In the "hello-melange-apko" repo, please see the following section: https://github.com/chainguard-dev/hello-melange-apko#generate-apk-repo-indexes

This is currently an extra step that users running through these steps manually need to run:

docker run --rm -it -v $(pwd):/w -w /w/packages -v $(pwd)/packages:/w/packages \
    --entrypoint sh \
    distroless.dev/melange -c \
        'for d in `find . -type d -mindepth 1`; do \
            ( \
                cd $d && \
                apk index -o APKINDEX.tar.gz *.apk && \
                melange sign-index --signing-key=../../melange.rsa APKINDEX.tar.gz\
            ) \
        done'

We should add a command to handle this, or perhaps a flag that automatically creates and signs the index after the APKs are successfully built.

busybox doesn't install with --use-proot

Need to distill down to a simpler use case, but:

melange build examples/gnu-hello.yaml --arch x86_64,armv7,arm64 --empty-workspace --use-proot

Fails with error about symlinks when installing busybox

armv7 simple-hello fails with `chroot: can't execute '/bin/busybox': Exec format error`

Running from https://github.com/chainguard-dev/melange/tree/9e7b082649e3e36b81dfb84514c733e89ae8b494/examples/simple-hello

docker run --privileged --rm -v "${PWD}":/work   cgr.dev/chainguard/melange build melange.yaml   --arch armv7   --signing-key melange.rsa
2022/09/28 14:13:02 building for [arm/v7]
2022/09/28 14:13:02 melange (hello/armv7): melange is building:
2022/09/28 14:13:02 melange (hello/armv7):   configuration file: melange.yaml
2022/09/28 14:13:02 melange (hello/armv7):   workspace dir: /tmp/melange-workspace-2193273048
2022/09/28 14:13:02 melange (hello/armv7): evaluating pipelines for package requirements
2022/09/28 14:13:02 melange (hello/armv7):   adding package "make" for pipeline "Run autoconf make"
2022/09/28 14:13:02 melange (hello/armv7):   adding package "make" for pipeline "Run autoconf make install"
2022/09/28 14:13:02 melange (hello/armv7):   adding package "binutils" for pipeline "Strip binaries"
2022/09/28 14:13:02 melange (hello/armv7):   adding package "scanelf" for pipeline "Strip binaries"
2022/09/28 14:13:02 melange (hello/armv7): building workspace in '/tmp/melange-guest-3090450258' with apko
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0mbuild context:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  working directory: /tmp/melange-guest-3090450258
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  tarball path: 
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  use proot: false
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  source date: 0001-01-01 00:00:00 +0000 UTC
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  Docker mediatypes: false
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  SBOM output path: 
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  arch: armv7
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0mimage configuration:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  contents:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    repositories: [https://dl-cdn.alpinelinux.org/alpine/edge/main]
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    keyring:      []
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    packages:     [alpine-baselayout-data binutils build-base busybox ca-certificates-bundle make scanelf ssl_client]
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m  accounts:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    runas:  
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    users:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m      - uid=1000(build) gid=1000
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m    groups:
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0m      - gid=1000(build) members=[build]
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0mdoing pre-flight checks
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0mbuilding image fileystem in /tmp/melange-guest-3090450258
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] �[0minitializing apk database
Sep 28 14:13:02.480�[36m [INFO] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0mrunning: /sbin/apk add --initdb --arch armv7 --root /tmp/melange-guest-3090450258
Sep 28 14:13:02.483�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0mOK: 0 MiB in 0 packages
Sep 28 14:13:02.484�[36m [INFO] [arch:armv7] �[0minitializing apk world
Sep 28 14:13:02.484�[36m [INFO] [arch:armv7] �[0minitializing apk keyring
Sep 28 14:13:02.484�[37m [DEBUG] [arch:armv7] �[0minstalling key /usr/share/apk/keys/armv7/[email protected]
Sep 28 14:13:02.484�[36m [INFO] [arch:armv7] �[0minitializing apk repositories
Sep 28 14:13:02.484�[37m [DEBUG] [arch:armv7] �[0minstalling key /usr/share/apk/keys/armv7/[email protected]
Sep 28 14:13:02.484�[36m [INFO] [arch:armv7] �[0msynchronizing with desired apk world
Sep 28 14:13:02.484�[36m [INFO] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0mrunning: /sbin/apk fix --root /tmp/melange-guest-3090450258 --no-scripts --no-cache --update-cache --arch armv7
Sep 28 14:13:02.487�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0mfetch https://dl-cdn.alpinelinux.org/alpine/edge/main/armv7/APKINDEX.tar.gz
Sep 28 14:13:02.947�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(1/29) Installing alpine-baselayout-data (3.3.0-r2)
Sep 28 14:13:02.981�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(2/29) Installing musl (1.2.3-r1)
Sep 28 14:13:03.045�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(3/29) Installing libgcc (12.1.1_git20220630-r6)
Sep 28 14:13:03.066�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(4/29) Installing libstdc++ (12.1.1_git20220630-r6)
Sep 28 14:13:03.191�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(5/29) Installing zlib (1.2.12-r3)
Sep 28 14:13:03.218�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(6/29) Installing binutils (2.39-r2)
Sep 28 14:13:03.455�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(7/29) Installing libmagic (5.42-r0)
Sep 28 14:13:03.519�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(8/29) Installing file (5.42-r0)
Sep 28 14:13:03.543�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(9/29) Installing libgomp (12.1.1_git20220630-r6)
Sep 28 14:13:03.576�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(10/29) Installing libatomic (12.1.1_git20220630-r6)
Sep 28 14:13:03.599�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(11/29) Installing gmp (6.2.1-r2)
Sep 28 14:13:03.660�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(12/29) Installing isl25 (0.25-r0)
Sep 28 14:13:03.805�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(13/29) Installing mpfr4 (4.1.0-r0)
Sep 28 14:13:03.855�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(14/29) Installing mpc1 (1.2.1-r1)
Sep 28 14:13:03.881�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(15/29) Installing gcc (12.1.1_git20220630-r6)
Sep 28 14:13:07.321�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(16/29) Installing libstdc++-dev (12.1.1_git20220630-r6)
Sep 28 14:13:08.662�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(17/29) Installing musl-dev (1.2.3-r1)
Sep 28 14:13:08.892�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(18/29) Installing libc-dev (0.7.2-r3)
Sep 28 14:13:08.910�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(19/29) Installing g++ (12.1.1_git20220630-r6)
Sep 28 14:13:09.988�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(20/29) Installing make (4.3-r0)
Sep 28 14:13:10.039�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(21/29) Installing fortify-headers (1.1-r1)
Sep 28 14:13:10.069�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(22/29) Installing patch (2.7.6-r7)
Sep 28 14:13:10.105�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(23/29) Installing build-base (0.5-r3)
Sep 28 14:13:10.135�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(24/29) Installing busybox (1.35.0-r25)
Sep 28 14:13:10.218�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(25/29) Installing ca-certificates-bundle (20220614-r2)
Sep 28 14:13:10.265�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(26/29) Installing scanelf (1.3.5-r1)
Sep 28 14:13:10.296�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(27/29) Installing libcrypto3 (3.0.5-r2)
Sep 28 14:13:10.471�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(28/29) Installing libssl3 (3.0.5-r2)
Sep 28 14:13:10.517�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0m(29/29) Installing ssl_client (1.35.0-r25)
Sep 28 14:13:10.538�[37m [DEBUG] [arch:armv7] [cmd:apk] [use-proot:false] [use-qemu:] �[0mOK: 164 MiB in 29 packages
Sep 28 14:13:10.548�[36m [INFO] [arch:armv7] �[0mcreating user 1000(build)
Sep 28 14:13:10.549�[36m [INFO] [arch:armv7] �[0mcreating group 1000(build)
Sep 28 14:13:10.550�[36m [INFO] [arch:armv7] [cmd:/bin/busybox] [use-proot:false] [use-qemu:] �[0mrunning: /usr/sbin/chroot /tmp/melange-guest-3090450258 /bin/busybox --install -s
Sep 28 14:13:10.554�[37m [DEBUG] [arch:armv7] [cmd:/bin/busybox] [use-proot:false] [use-qemu:] �[0mchroot: can't execute '/bin/busybox': Exec format error
2022/09/28 14:13:10 ERROR: failed to build package. the build environment has been preserved:
2022/09/28 14:13:10 melange (hello/armv7):   workspace dir: /tmp/melange-workspace-2193273048
2022/09/28 14:13:10 melange (hello/armv7):   guest dir: /tmp/melange-guest-3090450258
Error: failed to build package: unable to build workspace: unable to generate image: failed to install busybox symlinks: failed to install busybox symlinks: exit status 126
2022/09/28 14:13:10 error during command execution: failed to build package: unable to build workspace: unable to generate image: failed to install busybox symlinks: failed to install busybox symlinks: exit status 126

Create melange development image

We need an easy-to-use melange development image, so that people can kick the tires without installing an Alpine VM.

This has been started at github.com/distroless/melange

Ability to package third-party created SBOMs

For a Go app, for example, I would like to use something such as bom to generate an SBOM, and package this along with the apk package. Later, in apko, would like to extract this and publish to an OCI registry

Building for multiple archs simultaneously fails

Interestingly this works for me:

melange build --arch x86_64 --empty-workspace melange.yaml

and

melange build --arch arm64 --empty-workspace melange.yaml

But this fails:

melange build --arch x86_64,arm64 --empty-workspace melange.yaml

I'm trying to build the distroless/melange repo.

I think it's probably to do with needed to run in separate workspaces, I see in the output:

2022/07/01 14:12:18 melange (melange/x86_64): fatal: destination path '.' already exists and is not an empty directory.

Declarative moving of files into subpackages

A lot of the time we will write custom pipelines like:

subpackages:
  - name: "foo"
    pipeline:
      - runs: |
         mkdir -p "${{targets.subpkgdir}}"/bin
         mv "${{targets.destdir}}"/bin/whatever "${{targets.subpkgdir}}"/bin

It would be cool if this could be done declaratively instead.

caching support

When using docker we often setup a cache for things to download(eg cargo metadata & packages) and sometimes things that are compiled eg rust libs etc since rust can incrementally rebuild based on only what needs to be rebuilt. An example would be'

RUN --mount=type=cache,target=/usr/local/cargo/registry \
    --mount=type=cache,target=/app/target \
    cargo build --release

This allows for large projects to be build in minutes, especially when cross compiling for non-native archs(eg aarch64 on amd64) normally due having to clean build from scratch this cause build times to be almost half a day on medium/large size rust projects when cross compiling.

Lockfiles (like `.melange.lock`)

hi!

It would be cool to have lockfiles like .melange.lock that work similar to Cargo.lock.

With a configuration like this:

environment:
  contents:
    repositories:
      - https://dl-cdn.alpinelinux.org/alpine/edge/main
      - https://dl-cdn.alpinelinux.org/alpine/edge/community
    packages:
      - alpine-baselayout-data
      - busybox
      - make
      - go
      - git
      - ca-certificates-bundle

The lockfile would contain the resolved versions for make, go, git, etc and help achieve reproduce builds even if one of the packages was updated in alpine in the meantime.

The file could be generated with a command like melange update and toml would probably be a good fit since it works well in git diff and has good machine readability.

Idea: melange init

It'd be cool to have a command to scaffold out a basic melange.yaml file. As a strawman, a basic melange init could emit a file containing:

package:
  name: TODO
  version: TODO
  epoch: 0
  description: TODO
  copyright:
    - license: TODO
  dependencies:
    runtime:
     # - TODO

environment:
  contents:
    repositories:
     # - TODO
    keyring:
     # - TODO
    packages:
     # - TODO

pipeline:
 # - name: TODO
 #   runs: TODO

This could be expanded with options like melange init --package-repo=alpine or wolfi, which would populate the .environment.contents.repositories and keyring with those values.

(If this sounds good we could also consider an apko init, with similar --package-repo flag, and a --nonroot flag to spit out the nonroot user boilerplate)

cc @eddiezane who actually had this idea

CVE-2021-41190: OCI Manifest Type Confusion Issue (Low Severity)

NOTE: This is based solely on a Github dependabot alert, so it's possible there is a false positive here.

Impact

Systems that rely on digest equivalence for image attestations may be vulnerable to type confusion. The actual impact within melange isn't known, but it is quite probable that it is not actually vulnerable at all.

Patches

Since this was a specification-level bug rather than implementation-specific, the patch depends on the library used. Affected libraries include:

To see the specific resolution recommended for melange, see: https://github.com/chainguard-dev/melange/security/dependabot

NOTE: You may need admin rights for the repository to see the details.

Background

This is a low-severity CVE, but given that it is related to image metadata, we should aim to do an above-average job with it.

References

use --pipeline-dir flag to extend the builtin ones not for overriding them

We (w/@Dentrax) thought of an idea to increase pipeline configs' UX because the --pipelines-dir flag overrides the built-in pipelines (which is /usr/share/melange/pipelines).

Suppose we are about to create new additional pipelines. In that case, we lose the default built-in pipelines. One workaround could be extending the custom pipelines directory by copying the built-in ones stored in the melange repository.

So, it'd be nice to extend the built-in pipelines with the pipelines given by the --pipeline-dir flag instead of doing workarounds.

WDYT @kaniini?

Consider alternatives to document-wide templating

Melange supports templating using Go's template syntax, which I think is really good. I was surprised however to find that templating was applied before parsing a melange.yaml file as YAML.

On the plus side, this means Melange supports meta-programming like this, which creates a subpackage per locale/language-name combo.

This also means that it's possible in theory to template anywhere in a Melange config, including keys:

package:
  name: description
  ${{package.name}}: lolwut

The downside of this is that a melange.yaml file on disk cannot be guaranteed to be parseable as YAML, which would let users leverage existing YAML tooling (yq, yamlfmt, schema validators, other YAML templating engines maybe, etc.), without having this built in to or otherwise supported by Melange.

I wonder if we should consider reducing the scope of the templating engine, and only have templates applied on (certain?) config values, and not document-wide.

The locale example above could be supported by relying on a shared pipeline (hard in this case because it's iteratively generating subpackages and not just pipeline steps), or a separate YAML-generation step that lives in wolfi-dev/os that gets checked in CI.

This would guarantee that Melange YAMLs are valid YAMLs, and let us leverage existing YAML tooling instead of having to invest it in it Melange itself.

If there's a broader thinking behind Melange's templating being done this way, that's fine, and that context would be helpful too -- it's totally possible that such powerful templates are more valuable than having parseable YAML.

I thought I'd bring this up pre-0.2 in case we decide to change course, since post-0.2 it becomes harder to put the genie back in the bottle.

Add epoch info to FAQ/README

We set the epoch to 1970 so we can create reproducible builds, we should add some information to the FAQ for places where this might break something. For example, urlwatch (Python) depends on PyYAML, which throws some assertion, if the epoch is before 1980.

.melangeignore behaviour

I thought putting the following in .melangeignore would be enough to ignore the git directory:

.git

But it still picks files underneath the .git directory. This does work:

.git/**

But I don't think it should be necessary, isn't the first one in keeping with .dockerignore etc?

Melange getting started docs

We really need some simple docs that explain:

  • what exactly melange is
  • why it's needed
  • how it works
  • how to get started

Clean workspace by default

It can be confusing to run melange twice and get different results, because there are already files in the workspace directory. This will happen whether or not --empty-workspace is used.

I think the best option is to have each run in a clean tmp directory unless otherwise specified. Another option would be to wipe the contents of workspace before a new run.

Verify melange.yaml files

Just had a bunch of weird errors because the first line package: was missing from the YAML. It would be a better experience if melange rejected this as "invalid yaml" (and even better if it told me what was wrong).

SBOM: Add minimal SBOM generator to melange

We need to build a minimal SBOM generator into melange. The initial POC of the generator should be able to take the information from the OpenSBOM parsers and render a valid document with them.

Add way to generate VEX data for a package

Each package (e.g. in Wolfi) should be able to have a corresponding VEX document.

We can generate that VEX document with Melange, by leveraging the secfixes data from the Melange config file. To do this, we just need to ensure that any data we'd use in the VEX document is available in the package's Melange config or the final apk itself.

Note: This means we probably need to extend the secfixes data structure to include additional VEX-related information, like impact and justfication.

Proposal: Add a new vex subcommand to melange. Something like this...

$ melange vex ./config.yaml
{
  ...VEX JSON document...
}

Rename --empty-directory

The --empty-directory name doesn't feel like it captures what the feature does. Maybe --no-copy, but that doesn't feel much better. I wonder if it would be better not to copy anything and add a flag to do it?

We could also use the docker term "context" so --no-context or --use-context. WDYT?

Naming is hard.

file name too long

I got the following error for the simple-hello example while trying to build an apk pkg for it:

$ docker container run --rm -ti --privileged golang:1.18-alpine sh
$ apk add git make bubblewrap libc-dev 
$ git clone https://github.com/chainguard-dev/melange.git
$ make melange
$ mv melange /usr/local/bin
$ cd examples/simple-hello
$ mkdir -p /usr/share/melange/pipelines
$ cp -R ../../pipelines /usr/share/melange
$ melange build melange.yaml 

Screen Shot 2022-06-27 at 7 35 33 PM

Do not allow multiarch build without qemu detected

Results in error such as

2022/07/05 23:28:50 melange (hello-doc/x86): warning: read |0: file already closed
...
Error: failed to build package: unable to run pipeline: exit status 77
2022/07/05 23:28:50 error during command execution: failed to build package: unable to run pipeline: exit status 77

Caching/Resume builds

It can take a long time to run a build, especially for larger projects. Often writing a melange file will require making changes and tweaks to later stages, it would be great to be able to restart a build from a previous point that having to restart from scratch.

Apko "environment" key-pairs do not persist in melange build

Example melange.yaml:

...
environment:
  contents:
    ...
  environment:
    ICU_DATA_FILTER_FILE: "./data-filter-en.json"
...
pipeline:
  - name: Check env
    runs: |
      env

Expecting output of env command to contain ICU_DATA_FILTER_FILE=./data-filter-en.json, but it does not

ensure monitorPipe goroutines are joined before terminating a pipeline

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x4ed93e]

goroutine 9 [running]:
log.(*Logger).Printf(0xc000405f20?, {0xbb39ce?, 0x2e?}, {0xc000405ef8?, 0x0?, 0x0?})
        log/log.go:205 +0x1e
chainguard.dev/melange/pkg/build.(*Pipeline).monitorPipe(0xc000342100, {0xccca80?, 0xc000012140})
        chainguard.dev/melange/pkg/build/pipeline.go:200 +0x1cf
created by chainguard.dev/melange/pkg/build.(*Pipeline).evalRun
        chainguard.dev/melange/pkg/build/pipeline.go:232 +0x357

Bring back the ability to do hermetic builds

When multiarch support was merged, it cost us the ability to do hermetic builds. Restore this ability by copying the directory the .melange.yaml file is in to the workspace. I guess we will want something like .melangeignore (as Docker has .dockerignore).

Docs: Clarify use-cases and scope

Per user feedback, some people have tried using other package manager instead of compiling from source, suggest updating the README with more clarity. Here is one suggestion (feel free to use or modify):

Original: "Build apk packages using declarative pipelines."
Modified: "Compile apk packages from source using declarative pipelines. This project is designed to build reproducible apk packages decoratively from source code using any native (make, PyBuilder, mvn, etc) tooling. Using other package managers (such as pip, npm) to generate an APK is out of scope and may result in errors.

Update Installation Dependencies

Even though the README install instructions run inside of a container, they run as privileged and leverage workstation dependencies. On Fedora I had to install:
dnf install qemu-user-static for instance to get my build working. Need to do some testing in other environments to see what the requirements are. However, could we just make a note in the README for now?

Embed built-in pipelines

Melange's built-in pipelines (e.g., fetch, patch, etc.) are defined here, and copied into /usr/share/melange/pipelines when you run make install, or available in that location when you use the Melange container image.

To make it easier to use Melange without considering shared state on a filesystem, and to avoid hacks where one user updates /usr/share/melange/pipelines/fetch.yaml to mine crypto or whatever, we should embed these pipeline definitions at build-time using go:embed.

We might want to have a release or two where both the shared dir and compiled-in embedded definitions are considered to avoid breaking, then remove support for /usr/share/melange/pipelines in a subsequent release.

Deduce and provide GNU triplets for target

In a lot of cases, it is nice to be able to provide x86_64-unknown-linux-gnu or equivalent triplet to build systems. GNU Autoconf, CMake, Meson and cargo use these triplets to determine how to build things.

Melange should try to deduce the correct triplet when possible for an environment and provide it as a macro, perhaps something like ${{host.triplet}}.

Default substitutions are a little unforgiving

The default substitutions listed here, and variables defined by with:s, require exactly the ${{foo.bar}} form of the variable, e.g., ${{targets.destdir}}, ${{inputs.uri}}.

That could be confusing for users coming from GitHub Actions, which uses a similar placeholder style, but accepts forms with more relaxed spacing, like ${{ github.token }}, ${{ github.token}}, etc.

We might want to relax our replacer logic to accept and trim spaces when matching. I don't know how hard that is practically.

Current replacer logic (as of now) is here:

return strings.NewReplacer(replacements...)

automatic detection of so provides checks /opt.

I'm currently working on packaging pypy binaries to bootstrap it and I found an issue with melange's so detection. Pypy has to have certain so files relative to it eg libtinfow.so.6 and libffi.so.6 hence it is usually packaged in /opt with symlinks to bin. However melange detects these as provides as well and causes conflicts prevent building pypy from source using its binaries.

Build with runtime python but pip could not find pip

I'm trying to build an apk with a client tool (pulsar-client), however, I would receive the following error when I run the code with

docker run --privileged --rm -v "${PWD}":/work \
  cgr.dev/chainguard/melange build python-melange.yaml \
  --arch x86,amd64,aarch64,armv7 \
  --signing-key melange.rsa

Exception : importlib.metadata.PackageNotFoundError: No package metadata was found for pip

The file is:

package:
  name: pulsar-python-client
  version: 0.1.0
  description: An custom apk packaing including pulsar python client
  target-architecture:
    - all
  copyright:
    - license: Apache-2.0
      paths:
        - "*"
  dependencies:
    runtime:
      - python3

environment:
  contents:
    repositories:
      - https://dl-cdn.alpinelinux.org/alpine/edge/main
      - https://dl-cdn.alpinelinux.org/alpine/edge/community
    packages:
      - alpine-baselayout-data
      - ca-certificates-bundle
      - busybox
      - gcc
      - musl-dev
      - python3
      - python3-dev
      - py3-pip
      - py3-virtualenv

pipeline:
  - name: install python client
    runs: |
      set -x
      pip install pulsar-client==2.10.2
      pip install pulsar-client[avro]=='2.10.2'
      pip install pulsar-client[functions]=='2.10.2'
      pip install pulsar-client[all]=='2.10.2'

I basically copied the code from https://github.com/chainguard-dev/hello-melange-apko/blob/main/py/melange.yaml, but it's not working.

Can we have an example how to install python?

question: what's required to build multi-arch with melange and github actions

i suppose i need qemu setup. so i use the following

- name: install qemu-user
   runs: apt install qemu-user

but then i still get this error. all is good if i build for amd64 only.

Error: failed to build package: unable to populate workspace: open /home/runner/work/test-repo/test-repo/workspace/x86_64/.git/objects/0e/a4be1961454522ac2eafb79b3da05b64627ea5: permission denied
839
2022/06/15 06:42:46 error during command execution: failed to build package: unable to populate workspace: open /home/runner/work/test-repo/test-repo/workspace/x86_64/.git/objects/0e/a4be1961454522ac2eafb79b3da05b64627ea5: permission denied
840

another question: to build multi-arch, i need to config both melange and apko right?

host arch bwrap: execvp /bin/sh: No such file or directory

I'm currently trying to build a rust project using melange as a apk for apko. After it installs everything and reaches the runs section the following error occurs

2022/10/07 13:05:53 melange (Lunas-blog/x86_64): bwrap: execvp /bin/sh: No such file or directory
2022/10/07 13:05:53 melange (Lunas-blog/x86_64): warning: read |0: file already closed

I originally ran into this with foreign arch's such as aarch64. I've already got systemd-binfmt setup. I tried using the recommended command.

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes 

but that doesn't change the output, I tried limiting to the host arch amd64 but the same error occurs regardless. I've tried using podman instead of docker but that changes nothing

Implement `melange import` command

The melange import command will consume a binary artifact (tarball, debian package, RPM, ...) and wrap it as an apk. This is intended for use with compatibility with legacy build systems / artifacts, to allow them to be managed as apks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.