buildpacks / imgutil

Helpful utilities for working with images

imgutil

Helpful utilities for working with images

Development

To format:

$ make format

To run tests:

$ make test

imgutil's Issues

local img.Save() should fail with a helpful error if base image does not exist

Today if you do the following:

local.NewImage("some-image", dockerClient, local.FromBaseImage("some-base-image"))

and some-base-image doesn't exist on the daemon, the image will fail to save on Windows with a cryptic error:

Failed to save image:failed to write image to the following tags: [lifecycle:101f194-dirty: image load 'lifecycle:101f194-dirty'. first error: embedded daemon response: re-exec error: exit status 1: output: ProcessBaseLayer \\?\C:\ProgramData\Docker\windowsfilter\49e547ec94502d3e8d1e8de5be79a631f6247fb6fadb57bf288c4c22981a84af: Cannot create a file when that file already exists.] exit status 1

Through experimentation, it seems like the image will actually save successfully on Linux, though I am not sure how this works and why it is different.

pack and lifecycle/tools/image/main.go currently circumvent this issue by pulling the base image prior to initializing the new image.

Outcome

imgutil should produce a helpful error in this scenario.

SetLabel panics if a local image has no labels

Change to support Windows and Linux images

Prerequisite for buildpacks/pack#469

WindowsWriter needs to add `Files` prefix to links

At present, any link added to a Windows layer entry gets Files prepended to the header.Name but not to the header.Linkname meaning the expanded file would be invalid.

This WindowsWriter should be changed to prepend Files to header.Linkname as well as header.Name.

Related implementation:
https://github.com/moby/buildkit/blob/56e2ea083a4a392da9fb3a25a9e7cae34430948e/util/winlayers/differ.go#L245-L247

Create an archive package in imgutil that exposes the ReadXAsTar functions

Please see related PR and discussion: #73

Expose a method to parse an image reference into a local path

Reason

As part of the implementation of the export to OCI Layout feature we required to translate an image reference and map it to a path in the local filesystem. This operation is useful for Lifecycle implementors but also for Platform implementors, that's why the best place to offer this utility method is on imgUtil repository.

Expected behavior

Considering an image reference refers to either a tag reference or digest reference. It has the following formats

• A tag reference refers to an identifier of form <registry>/<repo>:<tag>
• A digest reference refers to a content addressable identifier of form <registry>/<repo>@<algorithm>:<digest>

Expose a method like:

classDiagram

class layout {
    ParseRefToPath(imageRef string) (string, error)
}

This method parses the given image reference to local path directory following the rules:

• WHEN the image reference points to a tag reference returns <registry>/<repo>/<tag>
• WHEN the image reference points to a digest reference returns <registry>/<repo>/<algorithm>/<digest>

Related

See issue 152 for more context

Add annotation `org.opencontainers.image.ref.name` when image in OCI layout is saved

Description

As part of the discussion of the export to OCI layout RFC we require to add the org.opencontainers.image.ref.name annotation to the final OCI layout image when it is saved.

The reason to that is because other tools in the ecosystem uses or add that annotation. For example:

• umoci is a tool used to create a runtime bundle from an image in OCI layout format, it requires the annotation org.opencontainers.image.ref.name to be present.
• skopeo tool include the annotation when an image is copy in OCI layout format.

Acceptance Criteria

When an image is created using the layout.NewImage(...) method the resulting image on disk after Save(), SaveAs() is called should contain the org.opencontainers.image.ref.name annotation as described in the spec

Implement the SaveFile() method in the layout image interface

Saves an OCI layout image on disk as a tarball

LocalImage should return a DigestIdentifier if available

Given a LocalImage created from a tag or digest reference, the returned Identifier should be a DigestIdentifier if the docker inspect contains a RepoDigest with repository matching the repository from the original requested reference.

Note While implementing this we might want to improve the workflow for fetching identifiers. Identifiers should only be available for saved images, not those that have been mutated but not yet commited.

Revert docker version upgrade

The recent update of the docker/docker lib #68 has created negative cascading effects in regards to how pack can import lifecycle + imgutil + docker libraries (plus other dependencies that update golang's sys package).

Prior to said PR, we were using v1.4.2-0.20190924003213-a8608b5b67c7 (the same version as GGCR).

At a high-level, most of the issues stem from that fact that docker/docker (aka moby/moby aka docker/engine 😓) is not go module friendly...

Relevant conversations:

• moby/moby#41503
• moby/moby#39302
• moby/moby#39056

...making some lower level dependencies trail off and cause conflicts such as:

• buildpacks/pack#901 (comment)
• buildpacks/pack#895 (comment)

Various attempts are resolving this issue have yield erratically unexpected failures (primarily affecting Windows).

Solutions attempted:

• Pinning to sys package of upgraded docker version: ory/dockertest#208 (comment)
• (Just 'cause) Pointing to docker/engine: moby/moby#39302 (comment)
• Pinning to docker/docker@master

I don't propose we indefinitely remain at the prior version but we should be more strategic in the upgrade based on our current dependency structure. I suspect that pack ran into this issue sooner rather than later but other platforms may find similar issues.

I would propose that we test and coordinate the upgrade in the following order GGCR -> imgutil -> lifecycle -> pack.

Support Saving images as archives.

Summary

We would like to be able to call img.SaveAsFile(filepath) (or something similar) and have our images saved as some sort of archive.

This ask has popped up in a couple of places:
buildpacks/pack#1125
buildpacks/lifecycle#423
buildpacks/pack#768 (needed for some of the exporter changes discussed here)

Use default compression for remote image AddLayer

Based on discoveries mentioned here (buildpacks/pack#351 (comment)), we need to use gzip.DefaultCompression when creating a tarball.Layer to simulate docker push compression and layer digest generation.

This could easily be addressed when google/go-containerregistry#574 is merged or if needed we could compress the tar file ourselves and use tarball.LayerFromOpener.

Should we limit the number of bytes in an image?

This question is spurred from conversation around this PR: #113 which attempts to use io.CopyN instead of io.Copy when untarring an image. Our use of io.Copy was flagged by muse-dev as being potentially vulnerable to DOS attacks.

In order for the change to io.CopyN to be meaningful, we should check the total number of bytes read as we are reading and throw an error if the number exceeds some threshold.

Some questions:

• Is DOS attack a realistic concern when running with a daemon? It seems the attacker would be DOS-ing their own machine, unless running in a cloud environment.
• If we decide to do this mitigation, what is a reasonable number to use for "max number of bytes"?

Expose a method to parse an image layout identifier

Reason

As part of the implementation of the export to OCI Layout feature the RFC defines the
analyzed.toml file will be updated to include the reference format in case of layout is being used.

[image]
  reference = "<image reference>"

[run-image]
  reference = "<image reference>"

[previous-image]
  reference = "<image reference>"

Where

• [image|run-image|previos-image].reference MUST be either:
  • A digest reference to an image in an OCI registry
  • The ID of an image in a docker daemon
  • The path to an image in OCI layout format

We required to translate an image reference with the format [path]@[digest] and map it to an Identifier. Because imgUtil defines the layout.Identifier() implementation the best place to offer this utility method is on imgUtil repository.

Expected behavior

Considering an image identifier with the format [path]@[digest]
Expose a method like:

classDiagram

class layout {
   ParseIdentifier(identifier string) (Identifier, error) 
}

Images created against docker daemon failed to get saved against remote Artifactory

Images created using the local package seem to experience errors when the remote package is used to save the previously created image against a remote artifactory. (When performing a remote.Write)

The artifactory returns the error MANIFEST_INVALID: manifest invalid;

Preliminary investigations make it seem like the Config files created are different for either image (remote vs local). What was a little weird is that we were able to use the docker CLI to push the image to artifactory but were unable to use GGCR to perform the same operation.
Additionally, performing an operation like mutate.CreatedAt on the image updated the config to become a "valid" one.

Let me know if I can provide any further information.

Add codecov support

As preparation for buildpacks/pack#814, I was hoping to get the test coverage of code that I port over to this repo, to ensure that it's sufficiently tested to be a library component, but I noticed that codecov (or a similar code coverage tool) isn't used.

If you could add, so I could have more reassurance that my code is sufficiently tested, and so users could have more confidence in the repo, that would be very helpful.

NewImage constructors should allow overriding default image properties

Right now, several image parameters get set by default on a new, empty image, which can only be overridden with Set*(). It would be convenient to be able to set these in the constructor.

The most useful constructor parameters for common image config properties would be:

• os - remote only, to override linux default
• os.version - local/remote, for setting compatible Windows kernel version.
• architecture - local/remote, to override amd64 default

This should also be considered along with #54 which will need a way to disambiguate manifest lists during FromBaseImage (also os, os.version, architecture, though these fields map to the manifest platform not the image config)

local should use ggcr v1.Daemon

ggcr's Daemon implementation appears to now support daemon.Image and daemon.Write and appears that we could use it for the local implementation. If the daemon package has feature-parity with remote, I believe we could get many benefits from using daemon:

• Enable daemons to "pull" non-container-images (ORAS) or otherwise unpullable images, by dynamically rewriting remote-image metadata and/or layers before saving to the daemon during a pull.
  • This looks like it will be important for enabling buildpackages on Windows, which now require a Microsoft-proprietary base-layer for them to be stored on the daemon.
• More shared code between remote and local - perhaps eventually a single implementation with flags for remote/local.
• Use an upstream implementation to which Docker is more directly concerned with supporting.

daemon package: https://github.com/google/go-containerregistry/tree/master/pkg/v1/daemon

imgutil images should be v1.Images

Currently, a layout.Image implements the v1.Image interface (because the underlying "base" image is embedded). With some small changes, remote.Image could follow this pattern. local.Image is trickier because of the daemon, but with some work we could assemble enough information to return a v1.ConfigFile, erroring if a manifest is requested. We'll need to be careful to distinguish what is the "saved" image vs the "working image" (e.g., after AddLayer is called the layer is added to the working image but not the image that's saved in the disk/registry/daemon) but if we do it right we can remove a bunch of duplication across the layout/remote/local packages and make the interface more intuitive to work with.

Create an new interface to handle Manifest Lists

Context

The OCI spec defines an Image Index concept to handle multiple manifests in an OCI image. Currently, we expose an Image interface that is consumed no matter the implementation (daemon, registry or layout), this is very convenient to decouple the implementation.

Requeriment

This issue suggests the definition and implementation of a new interface to expose the operations to deal with an Image Index .

Proposal

Note: I am using underscore to refer to a package, for example, foo_Bar referes to a Bar class in the foo package

classDiagram
    ManifestList <|-- remote_ManifestList
    ManifestList <|--  local_ManifestList 
    remote_ManifestList o-- ManifestListOption

    class ManifestListOption {
        +WithMediaTypes(requested imgutil.MediaTypes)
        +WithPath(path string)
    }

    class remote_ManifestList {
        +NewManifestList(repoName string, keychain authn.Keychain, ops []ManifestListOption) ManifestList 
    }

    class local_ManifestList {
         +NewManifestList(repoName string, path string, ops []ManifestListOption) ManifestList
    }

    class ManifestList {
       <<interface>>
        +Add(repoName string) error
        +Remove(repoName string) error
        +Save() error
    }

Notes

• remote_ManifestList must be added into the imgutil.remote package in a manifest_list.go file

Fallback to slow behavior if writing to daemon fails with performance optimization

We currently omit base image layers when writing to the docker daemon in order to improve performance. However some container runtimes like Podman that implement the daemon API do not allow an image load with missing layers. Therefore if our first attempt at writing fails, we should fall back on the slow/complete behavior and provide all layer tars.

context: buildpacks/pack#925

Add ability to override v1Config() function

The default is great, but it would be nice to be able to override the v1Config() function (or potentially wrap it) to inject platform-specific values into the image config. I'm happy to make the relevant PR(s) if others think this is a good idea :)

Continue with test cleanup if previous cleanup fails

In our tests, we have some cleanup steps that are wrapped in an h.AssertNil. If the cleanup fails, we'll halt execution and not do other cleanup steps.

If any cleanup steps fail, we should set the test outcome to failed, but we should continue with other cleanup. See conversation here: #114 (comment) for how pack accomplishes this.

Migrate `FileSystemLocalImage`to GGCR tarball image

This might be an opportunity to start migrating the local image to use ggcr. ggcr has a tarball type image that would work well for this https://godoc.org/github.com/google/go-containerregistry/pkg/v1/tarball. We don't need to block merging on this but I wanted to point it out.

Originally posted by @ekcasey in https://github.com/buildpack/imgutil/diffs/5

un-vendor go modules

Move tests to github actions

For consistency with other buildpacks repos

local.NewImage fails because docker.SaveImage returns multiple manifests.

Periodically when executing the lifecycle against the daemon errors like this occur:

ERROR: rebase working image: manifest.json had unexpected number of entries: 2

This happens when we are saving an image from the docker daemon to disk and we unexpectedly get more than one manifest.

imgutil/local/local.go

Lines 542 to 544 in afd98bd

	if len(manifest) != 1 {
	return nil, fmt.Errorf("manifest.json had unexpected number of entries: %d", len(manifest))
	}

We should figure out how we get into this state and determine a mitigation.

remote.NewImage() with remote.FromBaseImage() fails on Windows when provided a manifest list

E.g.,

remote.NewImage(tags[0], authn.DefaultKeychain, remote.FromBaseImage("mcr.microsoft.com/windows/nanoserver:1809") (a manifest list)

fails with

connect to repo store 'mcr.microsoft.com/windows/nanoserver:1809': no child with platform amd64/linux in index mcr.microsoft.com/windows/nanoserver:1809

Whereas

remote.NewImage(tags[0], authn.DefaultKeychain, remote.FromBaseImage("mcr.microsoft.com/windows/nanoserver:1809-amd64") (an image index)

works just fine.

WindowsWriter layers fail to initialize `MSWINDOWS.rawsd` when other `header.PAXRecords` exist

Existing, unrelated PAXRecords should not prevent MSWINDOWS.rawsd from being initialized. However, we should still only initialize MSWINDOWS.rawsd when it itself is not yet set.

imgutil/layer/windows_writer.go

Lines 42 to 45 in 7b9490c

	if header.PAXRecords == nil {
	header.Format = tar.FormatPAX
	header.PAXRecords = initializePaxPermissions(header)

Add annotation org.opencontainers.image.ref.name when Remote image is saved

TBD

Tests should support local and remote daemons

For a standard pack developer flow, I'd like to develop locally (on a MacOS workstation) and run tests quickly against a local daemon (e.g. Docker Desktop with Linux containers) and a remote daemon via DOCKER_HOST (e.g. local Windows VM running Docker Desktop with Windows containers). Other useful scenarios could be running tests against older daemons or Docker-in-Docker rootless daemons and be able to fix issues before committing/pushing.

This would also give some coverage for DOCKER_HOST functionality, which technically works though tests fail.

I'm interested in opening a PR to change the tests as follows:

• Change test image names from localhost:<registry.Port>/random-image-name to <DOCKER_HOST ip>:<registry.Port>/random-image-name only when DOCKER_HOST is set
• Change test registry containers to use HTTP Basic Auth, so CI containers won't be compromised - similar to pack's tests.
• Change remote to query using test registry Basic Auth credentials. Two possible approaches:
  • Indirectly through auth.DefaultKeychain by setting DOCKER_CONFIG to point to a test config with auths containing creds which default keychain loads.
  • Explicitly by replacing changing each remote.NewImage to use a test keychain instead that loads the credentials.
• Change remote to not push/pull to daemon, instead use go-containerregistry to directly work with manifests and images directly on the registry.

Fix performance penalty incurred by sending legacy formatted tars to the daemon

See here - if we can, we should send an OCI formatted tar to the daemon. This will be true when we have all the layers (from first doing a docker image save).

The changes can be made here. We should just stream an OCI layout-formatted tar to the daemon instead of building up the image in the old format.

Update GitHub Actions workflows to use environment files

Replace use of the now unsupported set-env command with environment files in all GHA tasks.

context:

• similar issue: buildpacks/lifecycle#424
• failing test: https://github.com/buildpacks/imgutil/runs/1451946342?check_suite_focus=true

local.Rebase/ReuseLayer leak temp files

It appears some internally generated/downloaded layers (note: temp files, not daemon layers) get left behind after Rebase and ReuseLayer. This does not impact the output images, just takes up disk space in the environment where these functions are called.

Repro (with pack rebase ... pack src -> lifecycle src):

git clone github.com/buildpacks/samples
cd samples
make build-alpine

mkdir tmpstuff
TMPDIR=$PWD/tmpstuff  pack rebase sample-kotlin-gradle-app:alpine

du -h -a tmpstuff
### OUTPUT ###
4.0K    tmpstuff//imgutil.local.image.032493284/a99d3f0802478557b0a44efae5e5a4928414594a9dd9ea69a8d1da181b89a9df.json
4.0K    tmpstuff//imgutil.local.image.032493284/6994ead759479b7875be89121e6e9f48910f8cdac4f57778d85302f465d0adaa/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/6994ead759479b7875be89121e6e9f48910f8cdac4f57778d85302f465d0adaa/json
4.0K    tmpstuff//imgutil.local.image.032493284/6994ead759479b7875be89121e6e9f48910f8cdac4f57778d85302f465d0adaa/VERSION
 12K    tmpstuff//imgutil.local.image.032493284/6994ead759479b7875be89121e6e9f48910f8cdac4f57778d85302f465d0adaa
2.7M    tmpstuff//imgutil.local.image.032493284/525f14e123a954c5ac257a0ab234f02e8e4204c067004ebc77aceaaef823eb96/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/525f14e123a954c5ac257a0ab234f02e8e4204c067004ebc77aceaaef823eb96/json
4.0K    tmpstuff//imgutil.local.image.032493284/525f14e123a954c5ac257a0ab234f02e8e4204c067004ebc77aceaaef823eb96/VERSION
2.7M    tmpstuff//imgutil.local.image.032493284/525f14e123a954c5ac257a0ab234f02e8e4204c067004ebc77aceaaef823eb96
 12K    tmpstuff//imgutil.local.image.032493284/86dee262a9b01683307b452211da1ca3754798bcaf8d6181187c45cc6be22a59/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/86dee262a9b01683307b452211da1ca3754798bcaf8d6181187c45cc6be22a59/json
4.0K    tmpstuff//imgutil.local.image.032493284/86dee262a9b01683307b452211da1ca3754798bcaf8d6181187c45cc6be22a59/VERSION
 20K    tmpstuff//imgutil.local.image.032493284/86dee262a9b01683307b452211da1ca3754798bcaf8d6181187c45cc6be22a59
4.0K    tmpstuff//imgutil.local.image.032493284/repositories
5.6M    tmpstuff//imgutil.local.image.032493284/bcf486159b1fbeb1b8d1a50fd7b9d46f97f2d59e6380fd9d788c889dade56eab/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/bcf486159b1fbeb1b8d1a50fd7b9d46f97f2d59e6380fd9d788c889dade56eab/json
4.0K    tmpstuff//imgutil.local.image.032493284/bcf486159b1fbeb1b8d1a50fd7b9d46f97f2d59e6380fd9d788c889dade56eab/VERSION
5.6M    tmpstuff//imgutil.local.image.032493284/bcf486159b1fbeb1b8d1a50fd7b9d46f97f2d59e6380fd9d788c889dade56eab
 22M    tmpstuff//imgutil.local.image.032493284/8e2746c2ab899aa80cb43ef1578fbe8f49a10005745c3a4a9d835ccc1d6d09de/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/8e2746c2ab899aa80cb43ef1578fbe8f49a10005745c3a4a9d835ccc1d6d09de/json
4.0K    tmpstuff//imgutil.local.image.032493284/8e2746c2ab899aa80cb43ef1578fbe8f49a10005745c3a4a9d835ccc1d6d09de/VERSION
 22M    tmpstuff//imgutil.local.image.032493284/8e2746c2ab899aa80cb43ef1578fbe8f49a10005745c3a4a9d835ccc1d6d09de
2.2M    tmpstuff//imgutil.local.image.032493284/94f0a64e124b02b14b5b2fa39992851f2bed27f5fed69841166ac960271eb8c4/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/94f0a64e124b02b14b5b2fa39992851f2bed27f5fed69841166ac960271eb8c4/json
4.0K    tmpstuff//imgutil.local.image.032493284/94f0a64e124b02b14b5b2fa39992851f2bed27f5fed69841166ac960271eb8c4/VERSION
2.2M    tmpstuff//imgutil.local.image.032493284/94f0a64e124b02b14b5b2fa39992851f2bed27f5fed69841166ac960271eb8c4
4.0K    tmpstuff//imgutil.local.image.032493284/manifest.json
201M    tmpstuff//imgutil.local.image.032493284/6d4e3d565d9c0c3e5225325cfec30fed5c06ae3603d4b57f4786d2002745e988/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/6d4e3d565d9c0c3e5225325cfec30fed5c06ae3603d4b57f4786d2002745e988/json
4.0K    tmpstuff//imgutil.local.image.032493284/6d4e3d565d9c0c3e5225325cfec30fed5c06ae3603d4b57f4786d2002745e988/VERSION
201M    tmpstuff//imgutil.local.image.032493284/6d4e3d565d9c0c3e5225325cfec30fed5c06ae3603d4b57f4786d2002745e988
4.0K    tmpstuff//imgutil.local.image.032493284/5c0a1c5355861df4fd73d0d0f31cf1765c4966b8fde64fa417ecc464d255d1c9/layer.tar
4.0K    tmpstuff//imgutil.local.image.032493284/5c0a1c5355861df4fd73d0d0f31cf1765c4966b8fde64fa417ecc464d255d1c9/json
4.0K    tmpstuff//imgutil.local.image.032493284/5c0a1c5355861df4fd73d0d0f31cf1765c4966b8fde64fa417ecc464d255d1c9/VERSION
 12K    tmpstuff//imgutil.local.image.032493284/5c0a1c5355861df4fd73d0d0f31cf1765c4966b8fde64fa417ecc464d255d1c9
233M    tmpstuff//imgutil.local.image.032493284
233M    tmpstuff/

Implement cnb.Image interface using OCI layout format

Expected Behavior

New package layout will be exposed with the implementation of the Image interface based on the OCI Image Layout

Note: There is a v1.Image wrapper implementation in the lifecycle that mimics the behavior of a Image without actually having the layers on disk.

The minimal implementation requires to expose the following classes:

classDiagram
class ImageOption {
    <<interface>>
    FromBaseImage(base v1.Image) ImageOption
    FromBaseImagePath(path string) ImageOption
    WithPreviousImage(path string)
}
class LayoutImage {
    NewImage(path string, ops ...ImageOption) (*Image, error)
}
class SparseImage {
     NewImage(path string, from v1.Image) (*Image, error) 
}

Where

• LayoutImage: is actually layout.Image but mermaid doesn't allow to define the package name. Implements the Image interface and handle the complexity of saving on Disk in OCI layout format.
• SparseImage: sparse.Image is an alternative implementation of the Image interface that mimics the behavior onf a image without actually saving any layer on disk.

fakes package should expose a way to create a non-existent image

See TODO in lifecycle: https://github.com/buildpacks/lifecycle/blob/277f3b2834ed210ddf34ff6c041157cd3efe328c/exporter_test.go#L833-L835