foundriesio / ci-scripts Goto Github PK

That looks nice to try and hide things. But it winds up producing a message like this:

ERROR:System Image Assembler:Failed to assemble a system image(s): 'image-file'

I'm assuming there was some KeyError thrown somewhere. But lacking the exception (and really stack trace) makes things like this hard to dig into.

The Foundries Factory currently has rudimentary support for managing tags that are included with OTA Targets as the result of a CI build. I'm opening this issue to explain the current state of tagging, what is missing, and then get an agreement on the solution.

Terminology

Platform Build - A build created by a change to the LMP/OE. This is the base OS image
Container Build - A build created by a change to containers.get.
Target - This a TUF entry in a factory's targets.json file. It represents what we should think of as an immutable combination of the Platform Build's OSTree hash + the output of a Container Build.
Tag - A Target has a "custom" section where we maintain a list of Tags. The tags can be used to say things like "this is a development build", or this is a "production" build.

What We Have Now

Currently our tagging logic assumes one Platform branch and one Containers branch are paired together. The way this works is as follows:

A change to the "devel" branch on containers.git would cause new docker containers to get built. It then creates a new Target with these containers referenced in the docker-apps section. A Target needs a Platform build to be complete, so it scans through targets.json to find the previous Target with the "devel" tag. It then copies its OSTree hash to produce the new Target.

A change to the "devel" Platform is similar. It produces a new Target with the OStree hash from its build. It then finds the previous "devel" Target in targets.json and copies the docker apps that are defined in it.

This is al done by having a single environment variable defined in CI: OTA_LITE_TAG.

New Needs

There are two new scenarios this logic can't cover:

We want to add a new "postmerge-stable" Platform build. However, we just want to use the current containers from the most recent "postmerge" merge.

This also means that a change to the Containers project would now need to create 2 Targets - a "postmerge" and "postmerge-stable".

There is a 3rd scenario users have: There are 2 Container tags that should be based on the same Platform. This turns out to really be the same as the previous two scenarios.

Proposed Solution

OTA_LITE_TAG isn't expressive enough. You actually need to express:

• This build can $X tags
• I'm tagging this with $X, but it should base its target on content from the previous $Y tag.

I think we could (ab)use the currrent OTA_LITE_TAG to express this. The format could be defined like:

TAG[:INHERIT_TAG][,TAG[:INHERIT_TAG]....]

How this addresses our scenarios:

postmerge-stable Platform build should grab latest postmerge Containers:
OTA_LITE_TAG: postmerge-stable:postmerge

Which means a change to containers.git produces 2 tags now (postmerge and postmerge-stable):
OTA_LITE_TAG: postmerge,postmerge-stable

One platform with multiple containers. Let's call things:

• Platform - devel-X
• Container - devel-Y
• Container - devel Z

So a Platform devel-X change could be expressed with:
OTA_LITE_TAG:devel-X,devel-Y

A change to either container stays "normal". ie its just OTA_LITE_TAG: devel-Y

It looks like the EXTRA_TAGS option may have been broken with the move to docker-buildx.

We need to add a generic way to pass docker --build-arg(s) to be used in the Dockerfile. To make things more clear I think build.sh should check for environment variables starting with DOCKER_ARG_ and pass them all as --build-arg arguments. eg:

 DOCKER_ARG_FOO=foo
 DOCKER_ARG_BAR=bar

Would add to our docker build command: --build-arg FOO=foo --build-arg BAR=bar

It will then be up to a factory's container.yml to pass the values they want into the CI job.

There are some variables that are important and should not have to be set via lmp.params, such as IMAGES and DOCKER_SECRETS. For example, if we do not define IMAGES then it will produce an error. My understanding is that lmp.params is intended to be a user-configured section, with only user additions. So it makes sense to separate these two, as currently we set:

lmp:
  params:
    IMAGE: lmp-factory-image

which encroaches upon the user's space. The same will be true if we introduce other variables here by default in future, instead of creating new directives for these variables.