estesp / mquery

Multi-platform (manifest list/OCI index) registry image query utility

mquery

A simple utility and serverless-based backend for querying Docker v2 & OCI v1 container images and reporting on "manifest list"/OCI "index" multi-platform image support.

About

This project uses AWS Lambda as a backend, in concert with the manifest-tool inspect API capability to easily report on the status of whether an image is a manifest list/OCI index entry in a registry, and if so, what platforms are supported by the image.

Usage

You can use the public endpoint with curl and JSON formatting tools to query images directly. See the next section for a tool which performs this for you and provides a simple text output. This tool is published as a multi-platform image on DockerHub as mplatform/mquery; for example you can look up the mplatform/mquery:latest image as follows:

$ docker run --rm mplatform/mquery mplatform/mquery:latest
Image: mplatform/mquery:latest (digest: sha256:d0989420b6f0d2b929fd9355f15c767f62d0e9a72cdf999d1eb16e6073782c71)
 * Manifest List: Yes (Image type: application/vnd.docker.distribution.manifest.list.v2+json)
 * Supported platforms:
   - linux/ppc64le
   - linux/amd64
   - linux/386
   - linux/s390x
   - linux/riscv64
   - linux/arm64/v8
   - linux/arm/v7
   - linux/arm/v6
   - windows/amd64:10.0.17763.2300
   - windows/amd64:10.0.14393.4770

Using the mquery tool

This project also includes a tool for querying the Lambda API Gateway-fronted endpoint with a simple/readable output format for showing the list of platforms supported by a specific image. You can build the tool yourself using the Makefile, or you can use a pre-packaged multi-platform image on DockerHub as shown in the section above.

This Go program requires the github.com/dghubble/sling and github.com/opencontainers/image-spec/specs-go/v1 packages. You can add these to your Go development environment with:

$ go get -u github.com/dghubble/sling
$ go get -u github.com/opencontainers/image-spec/specs-go/v1

References

More information about manifest lists and multi-platform image support is available in these blog posts:

• DockerHub Official Images Go Multi-platform! - 13 Sep 2017
• [Docker Blog] Docker Official Images are now Multi-platform - 19 Sep 2017
• A big step towards multi-platform Docker images - 22 April 2016

Also see the manifest-tool project for an easy to use tool for assembling and pushing manifest lists and OCI index images.

License

This project is licensed under the Apache Public License, v2.0.

mquery's Issues

Add all Windows OS variants to better support process isolation on Windows

Hey Phil,

what do you think about this idea. I would love to make the mquery tool available for all Windows OS variants. Only the Windows Server 2016 variant is listed in the manifest list of the mplatform/mquery image:

$ alias mq
alias mq='docker run --rm mplatform/mquery'

$ mq mplatform/mquery
Image: mplatform/mquery
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm
   - linux/arm64
   - linux/ppc64le
   - linux/s390x
   - windows/amd64:10.0.14393.1593

Well, I normally work on my MBP, but as Windows 10 now has process isolation to run Windows containers without the Hyper-V isolation it makes sense to have Windows images for the current Windows kernel version. Only then process isolation will work and people could use the mquery tool.

The golang image is an excellent example with all four Windows variants for 2016, 1709, 1803 and now 1809.

$ mq golang
Image: golang
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm/v7
   - linux/arm64
   - linux/386
   - linux/ppc64le
   - linux/s390x
   - windows/amd64:10.0.14393.2724
   - windows/amd64:10.0.16299.904
   - windows/amd64:10.0.17134.523
   - windows/amd64:10.0.17763.253

The technical challenge is that you normally need a Windows Docker host with the same or newer kernel than the Windows base image version. TL/DR you need a Windows Server 2019 or Windows 10, version 1809 to build for the latest 1809 nanoserver image. Older versions can be built in hyperv isolation mode.

I remember we have added an AppVeyor build some time ago, at the moment AppVeyor still only has Windows Server 2016 - the oldest variant.

But as a Captain this is not an impossible way. I wrote about a tool I use to create Windows images for all four variants with just AppVeyor. https://stefanscherer.github.io/poc-build-images-for-1709-without-1709/
This would work fine for this mquery tool. mquery is just a Golang binary in a nanoserver image. In this case I found out that it's fine to "rebase" the application layer on top of different Windows nanoserver images.

I do this with a multi-arch whoami service

$ mq stefanscherer/whoami
Image: stefanscherer/whoami
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm/v6
   - linux/arm64
   - windows/amd64:10.0.14393.2551
   - windows/amd64:10.0.16299.904
   - windows/amd64:10.0.17134.523
   - windows/amd64:10.0.17763.253

The complete AppVeyor build pipeline is available in https://github.com/StefanScherer/whoami

WDYT?

AWS lambda backend is unreachable

Hello. My queries are all failing:

docker run --rm mplatform/mquery mplatform/mquery:latest
ERROR: failed to query backend: Get "https://2xopp470jc.execute-api.us-east-2.amazonaws.com/mquery?image=mplatform%2Fmquery%3Alatest": dial tcp 18.191.128.67:443: connect: network is unreachable

Include name of image being queried in output

For ease of using mquery as input to a downstream function, add the name of the image being queried in the output of the tool.

arm64 shows as arm64/undefined

Based on containerd/platform implementation, variant should not be required for arm64 and missing in the normalized mode. But in mquery output it shows as arm64/undefined if missing.

> docker run --rm mplatform/mquery tonistiigi/copy:v0.1.5
Image: tonistiigi/copy:v0.1.5
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm64/undefined
   - linux/arm/v7
   - linux/ppc64le
   - linux/s390x

» go run dhub.go manifest tonistiigi/copy v0.1.5                                                               !10266
{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:622cb58662caba19119d8470e5cb355b08f2a94a4d30996108890f74197ffa54",
         "size": 1365,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:8a0ad02b8fa9a97bf472ac1779447d227b3369e22c67d22b742df805bd331ea3",
         "size": 1155,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:f05dc1efac3e9a2f2b71af1a66c87f0e2c1072e6cadce9d405b54a1204e7ea4c",
         "size": 1155,
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:2b09703b0299eea011cec989798bc2d107008417abe1ecc23484a7ea30e8b514",
         "size": 1155,
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:e197a4f3565a1ebec0f23258c3fc51d900d7cf982742d2ea9b1b9cf946fa7bd9",
         "size": 1155,
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Let's make this a multi-arch image, so everyone on every Docker platform can run and use it.

As suggested by Stefan, let's build and distribute this so that it's a multi-arch image.

(Not only would it be useful, but it would be a good template for building and distributing multiarch Go applications.)

riscv architecture support?

I am wondering if this project would like to support the riscv architecture.

RISC-V is an open standard instruction set architecture (ISA) based on established reduced instruction set computer (RISC) principles. Unlike most other ISA designs, the RISC-V ISA is provided under open source licenses that do not require fees to use. (from Wikipedia)

Golang supports cross-compiling risc-v already (GOARCH=riscv64/GOOS=linux), and looking into this project I think adding a new architecture would be quite achievable.

As an early riscv docker user, most if not all docker images does not support riscv yet, and being able to run this tool from a riscv docker system would be great to know what arch is supported and what is not.

Image not found or not a public image reference

probably a problem with new rate limiting on docker hub. I'm not sure
I get the following in my CI pipeline when building multiplatform image:
(https://github.com/dutchcoders/transfer.sh/runs/1656312425)

Run docker run --rm mplatform/mquery dutchcoders/transfer.sh:edge
  docker run --rm mplatform/mquery dutchcoders/transfer.sh:edge
  shell: /bin/bash -e {0}
Unable to find image 'mplatform/mquery:latest' locally
latest: Pulling from mplatform/mquery
db6020507de3: Pulling fs layer
f11a2bcbeb86: Pulling fs layer
f11a2bcbeb86: Verifying Checksum
f11a2bcbeb86: Download complete
db6020507de3: Download complete
db6020507de3: Pull complete
f11a2bcbeb86: Pull complete
Digest: sha256:e15189e3d6fbcee8a6ad2ef04c1ec80420ab0fdcf0d70408c0e914af80dfb107
Status: Downloaded newer image for mplatform/mquery:latest
ERROR: Image not found or not a public image reference
Error: Process completed with exit code 1.

Show (Windows) os.version as well

It would be nice to also see the OSVersion / os.version field in the output of mquery.

Today I have created a manifest list spec for the two Windows image version available, one image for Windows Server 2016 LTS channel, and another image for the Windows Server 1709 Semi-annual channel.

You only can run containers with an OS version that matches with the Docker host. (OK, there is also HyperV isolation, but not all cloud providers have nested hypervisors to activate HyperV).

The output of mquery just tells me that it is for amd64/windows.

$ docker run mplatform/mquery stefanscherer/curl-windows
Image: stefanscherer/curl-windows
 * Manifest List: Yes
 * Supported platforms:
   - amd64/windows
   - amd64/windows

Maybe with : or another /?

$ docker run mplatform/mquery stefanscherer/curl-windows
Image: stefanscherer/curl-windows
 * Manifest List: Yes
 * Supported platforms:
   - amd64/windows:10.0.14393.1770
   - amd64/windows:10.0.16299.19

Mquery not producing results if on Ubuntu Linux arm64 (AWS Graviton)

Hello, first I want to thank you for creating this tool!

I was trying to run the docker image of this project via docker on an AWS Graviton VM and found that it does not produce the expected output:

$ docker run --rm mplatform/mquery openjdk:11
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

$ docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:56 2020
 OS/Arch:           linux/arm64
 Context:           default
 Experimental:      true

Same run in a amd64 VM:

$ docker run --rm mplatform/mquery openjdk:11
Image: openjdk:11
 * Manifest List: Yes
 * Supported platforms:
   - linux/amd64
   - linux/arm64
   - windows/amd64:10.0.17763.1697
   - windows/amd64:10.0.14393.4169

$ docker version
Client: Docker Engine - Community
 Version:           20.10.2
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        2291f61
 Built:             Mon Dec 28 16:17:43 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

mquery images contain wrong architecture in image config

 # docker buildx imagetools inspect mplatform/mquery
Name:      docker.io/mplatform/mquery:latest
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:e15189e3d6fbcee8a6ad2ef04c1ec80420ab0fdcf0d70408c0e914af80dfb107

Manifests:
  Name:      docker.io/mplatform/mquery:latest@sha256:84cf13c9dc46373fdd0ab1bf5459da0b644d8367c6e7149c22979fc366190631
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64

  Name:      docker.io/mplatform/mquery:latest@sha256:f55b14d9b031d38637d8d277364902eba603f76b7a9c42adc02a33d5d86799a1
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm

  Name:      docker.io/mplatform/mquery:latest@sha256:7fee21210e3505df0a85bb19cd41a51fe1be5818d5d45f4ca42af232acf8b030
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

...

Pick arm64

 # docker pull docker.io/mplatform/mquery:latest@sha256:7fee21210e3505df0a85bb19cd41a51fe1be5818d5d45f4ca42af232acf8b030
...
 # docker inspect -f '{{json .Architecture}}' docker.io/mplatform/mquery:latest@sha256:7fee21210e3505df0a85bb19cd41a51fe1be5818d5d45f4ca42af232acf8b030
"amd64

Same for linux/arm image.

In Docker 20.10 this means users get the following warning when running:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Trouble getting architectures for k8s.gcr.io

Hi 👋

Came across this one and thought I'd let you know :)

$ docker run --rm mplatform/mquery k8s.gcr.io/pause:latest
Image: k8s.gcr.io/pause:latest (digest: sha256:a78c2d6208eff9b672de43f880093100050983047b7b0afe0217d3656e1b0d5f)
 * Manifest List: No (Image type: application/vnd.docker.distribution.manifest.v2+json)
 * Supports: linux/amd64

whereas

$ docker pull --platform linux/arm64/v8 k8s.gcr.io/pause:3.5
$ docker inspect k8s.gcr.io/pause:3.5 | grep Arch
        "Architecture": "arm64",

(source)