agent - build crossplatform image for 2.0.0.12 about ladder99

[MRIIOT]

https://github.com/Ladder99/ladder99/blob/main/shell/commands/build

https://github.com/mtconnect/cppagent/blob/master/Dockerfile.alpine

https://github.com/mtconnect/cppagent/blob/master/Dockerfile

docker run -it --rm --privileged tonistiigi/binfmt --install all

git clone https://github.com/mtconnect/cppagent_dev.git
cd cppagent_dev

docker login

docker buildx build \
    --platform linux/amd64,linux/arm64 \
    --tag mtconnect/agent:2.0.0.13_RC1 \
    --push \
    .

docker logout

[bburns]

this is what i used for buildx - ie build image for each platform separately, then combine them into one image. takes better advantage of docker cache in case one of the platforms fails -
https://docs.mriiot.com/en/internal/mriiot/ladder99/shell-dev#build-1

the build2 script here

ladder99/shell/commands/build2

Line 72 in bdeed1c

for arch in amd64 arm64 arm ; do

- will loop over the platforms and do this for l99 services - not tested yet

[MRIIOT]

Man, why do these take 6 hours? Is it CPU bound?

docker buidocker buildx build \
    --platform linux/amd64,linux/arm64 \
    --tag mtconnect/agent:2.0.0.13_RC1 \
    --push \
    .
[+] Building 25522.3s (38/38) FINISHED
 => [internal] load build definition from Dockerfile                                                0.0s
 => => transferring dockerfile: 4.25kB                                                              0.0s
 => [internal] load .dockerignore                                                                   0.0s
 => => transferring context: 344B                                                                   0.0s
 => [linux/arm64 internal] load metadata for docker.io/library/ubuntu:22.04                         0.4s
 => [linux/amd64 internal] load metadata for docker.io/library/ubuntu:22.04                         0.4s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                       0.0s
 => [internal] load build context                                                                   0.2s
 => => transferring context: 75.46kB                                                                0.2s
 => CACHED [linux/amd64 os 1/1] FROM docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f  0.0s
 => => resolve docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a9  0.0s
 => CACHED [linux/arm64 os 1/1] FROM docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f  0.0s
 => => resolve docker.io/library/ubuntu:22.04@sha256:27cb6e6ccef575a4698b66f5de06c7ecd61589132d5a9  0.0s
 => [linux/amd64 build 1/5] RUN apt-get clean   && apt-get update   && apt-get install -y   build  67.7s
 => CACHED [linux/amd64 release 1/7] RUN apt-get update && apt-get install -y ruby                  0.0s
 => CACHED [linux/amd64 release 2/7] RUN useradd --create-home agent                                0.0s
 => [linux/arm64 build 1/5] RUN apt-get clean   && apt-get update   && apt-get install -y   buil  432.7s
 => [linux/arm64 release 1/7] RUN apt-get update && apt-get install -y ruby                       205.8s
 => [linux/amd64 build 2/5] WORKDIR /root/agent                                                     0.4s
 => [linux/amd64 build 3/5] COPY . .                                                                1.1s
 => [linux/amd64 build 4/5] RUN conan export conan/mqtt_cpp   && conan export conan/mruby   && c  991.4s
 => [linux/arm64 release 2/7] RUN useradd --create-home agent                                       0.7s
 => [linux/arm64 build 2/5] WORKDIR /root/agent                                                     0.4s
 => [linux/arm64 build 3/5] COPY . .                                                                1.1s
 => [linux/arm64 build 4/5] RUN conan export conan/mqtt_cpp   && conan export conan/mruby   &&  14302.9s
 => [linux/amd64 build 5/5] RUN conan build . -bf build                                          1260.5s
 => [linux/amd64 release 3/7] COPY --chown=agent:agent --from=build /root/agent/build/bin/agent /u  0.1s
 => [linux/amd64 release 4/7] COPY --chown=agent:agent --from=build /root/agent/schemas /etc/mtcon  0.1s
 => [linux/amd64 release 5/7] COPY --chown=agent:agent --from=build /root/agent/simulator /etc/mtc  0.0s
 => [linux/amd64 release 6/7] COPY --chown=agent:agent --from=build /root/agent/styles /etc/mtconn  0.0s
 => [linux/amd64 release 7/7] WORKDIR /home/agent                                                   0.0s
 => [linux/arm64 build 5/5] RUN conan build . -bf build                                         10768.4s
 => [linux/arm64 release 3/7] COPY --chown=agent:agent --from=build /root/agent/build/bin/agent /u  0.2s
 => [linux/arm64 release 4/7] COPY --chown=agent:agent --from=build /root/agent/schemas /etc/mtcon  0.1s
 => [linux/arm64 release 5/7] COPY --chown=agent:agent --from=build /root/agent/simulator /etc/mtc  0.0s
 => [linux/arm64 release 6/7] COPY --chown=agent:agent --from=build /root/agent/styles /etc/mtconn  0.1s
 => [linux/arm64 release 7/7] WORKDIR /home/agent                                                   0.0s
 => ERROR exporting to image                                                                       15.1s
 => => exporting layers                                                                            14.2s
 => => exporting manifest sha256:db487a0a910ea0b5a71dec7c3d1afbd6bb5df4983b0f16f2998a24ad1f0261ed   0.0s
 => => exporting config sha256:974c4321a8b4f9807340e5584f7c97e5261a27e9106a3600a40fc405caee5e1d     0.0s
 => => exporting manifest sha256:82f6ac3345a32befde96febb0db801ecaa7e0bc0878bfd8576d31d09f25f0a28   0.0s
 => => exporting config sha256:e206403c3bae9b0632e9aaac27978f1b663fcc9fe8f49ffba43419445b6079b2     0.0s
 => => exporting manifest list sha256:0062e763a0cbb826de84d900348ce8ef51d35024489f753fb101abe7e08f  0.0s
 => => pushing layers                                                                               0.8s
 => [auth] mtconnect/agent:pull,push token for registry-1.docker.io                                 0.0s
 => [auth] mtconnect/agent:pull,push token for registry-1.docker.io                                 0.0s
 => [auth] mtconnect/agent:pull,push token for registry-1.docker.io                                 0.0s
 => [auth] mtconnect/agent:pull,push token for registry-1.docker.io                                 0.0s
 => [auth] mtconnect/agent:pull,push token for registry-1.docker.io                                 0.0s
------
 > exporting to image:
------
ERROR: failed to solve: server message: insufficient_scope: authorization failed

[MRIIOT]

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker buildx rm l99builder
docker buildx create --name l99builder --driver docker-container --use
docker buildx inspect --bootstrap

git clone https://github.com/mtconnect/cppagent_dev.git
cd cppagent_dev

docker login

docker buildx build -f Dockerfile \
    --platform linux/amd64,linux/arm64 \
    --tag ladder99/agent:2.0.0.13_RC1 \
    --tag ladder99/agent:latest \
    --push \
    .


docker logout

[bburns]

Man, why do these take 6 hours? Is it CPU bound?

The qemu arm emulation is slow - might be able to speed up if use more cpus but my mac would run out of memory -

https://github.com/mtconnect/cppagent_dev/blob/master/Dockerfile#L58-L61

[tukusejssirs]

@bburns, how about creating separate scripts in package.json to build for each supported platform without buildx (e.g. build:win64, build:linux64, build:arm7)? I understand that build for each platform on a single computer is harder and we might still need buildx (or something else) for that, however, building for a the current computer platform only is the most used use case IMHO, which is IMHO better done without buildx. WDYT, @MRIIOT, @bburns?

might be able to speed up if use more cpus but my mac would run out of memory

How about adding a command-line option with the number of CPUs to be used? Like make -j8 means that eight cores are to be used by the compiler.

[MRIIOT]

The only two archs we should need is linux/amd64 and linux/arm64, so having a dedicated computer per arch would make more sense. Images can still get tagged the same. The emulation is just slow.

But then again, we can run build nightly with emulation and just let it run. Either way.

https://hub.docker.com/layers/ladder99/agent/2.0.0.13_RC1/images/sha256-4a8beed2066ed6c042ec05a011882a4a269288c40d1366f0bcf6acbcc5c002e5?context=repo

[tukusejssirs]

2.0.0.13_RC1

Is that a different version from ladder99 apps (services) versioning (0.10.1)? Is that intentional or why is it different? 🤔

[MRIIOT]

It is the latest MTConnect Agent version. Not our code, but we use it.
If we had to build Grafana for our use case, would we version it 0.10.1 or whatever official Grafana version is?

[tukusejssirs]

Well, I suggest to version all apps (services) in this repository under one version number (see #188).

That said, I might have been too quick: we don’t have agent app under this repo, thus it might be okay. Moreover, in the Docker image home page you have noted that First version number is the MTConnect Agent version (eg 1.7.0.3), second is the Ladder99 Agent UI version (eg 0.1.2), however, it does not seem to be true, as there is no 0.10.1 in the the tag.

For third-party stuff, original version number is enough unless we modify it (then we need to somehow differentiate it).

[MRIIOT]

Yeah, I don't know where 0.10.1 is coming from. The XSLT has been part of Agent repo for some time so it should be just Agent's version.

[tukusejssirs]

Yeah, I don't know where 0.10.1 is coming from.

From this repo codebase, from package.json files (see the description of #188):

Currently, each microservice seems to be versioned separately:

# `version` value in `package.json` of each app (microservice)
services/meter/package.json     : 0.1.0
services/recorder/package.json  : 0.1.0
services/adapter/package.json   : 0.10.1
services/relay/package.json     : 0.10.1
services/simulator/package.json : 0.1.0
services/compiler/package.json  : 0.1.0

[bburns]

If we had to build Grafana for our use case, would we version it 0.10.1 or whatever official Grafana version is?

call it grafana--?

in the Docker image home page you have noted that First version number is the MTConnect Agent version (eg 1.7.0.3), second is the Ladder99 Agent UI version (eg 0.1.2), however, it does not seem to be true, as there is no 0.10.1 in the the tag.

that was from an old version when i was working on the agent xsl separately - it's part of the agent now