Versioning of Docker Images about docker

[maxjuergens]

I disagree with you here. If you bundle Pimcore with it, you cannot change code without installing it outside the image anyway. This is image is supposed to be a base-image for development and running Pimcore containerized.

If you need your code bundled into the image, that is up to you.

I know, just wanted to add it in case someone wants to think about it further.

But, if it's never going to happen, you basically have a PHP variant and not Pimcore, eg:

pimcore/php:latest
pimcore/php:8.1
pimcore/php:8.1.8-v1.0.0

pimcore/php:8.1-supervisor
pimcore/php:8.1.8-supervisor-v1.0.0

pimcore/php:8.1-debug (or pimcore/php:8.1-xdebug)
pimcore/php:8.1.8-debug-v1.0.0

---

Whatever, just to make it short: I'd prefer a lower case tag. 😃

[yariksheptykin]

This is an interesting problem and I'd like to share the way I think about it. Maybe this can help.

So the problem here is with (image) versioning. @alegatti, I like that you mention composer, because I believe they solve version management quite gracefully by using semver, and lock files. I think therefore that we can borrow these principles to solve our problem here.

Lets consider locking first. For some enviroments such as production having reproducible builds is a king. So we want to lock as much as we can. We use composer.lock to install exact package versions, we use git to use the exact source code version. As for docker we also want to use a precise base image to derive from. Docker has a concept of image digest for this. With a digest we can reference the exact image build. According to the documentation our Dockerfile should use some form of this:

FROM [--platform=<platform>] <image>[@<digest>] [AS <name>]

So the consumer of pimcore image who want to maximize controll over the base image should opt for including image digest into image ref name. This is just like composer.lock or git ref. So for locking we don't need to add build ids to the image tags. Docker takes care of those via digest.

But we obviously cannot lock ourselves forever. There will be updates, which are (mostly) a good thing. One strategy to deal with updates is to always use the latest beta untested version. This is the best way to spot and address problems immediately as they occur (just like in "continuous integration"). Just as it happened with xdebug 3.0.0 for you, @alegatti. You noticed the problem, addressed it and solved it immediately, which is good. We can of course let this happen only in safe environments such as local dev env.

When problems caused by updates however arise often they can become pretty annoying and distracting. One way to reduce the number of problems introduced with updates is to provide better testing before releasing an update. This can improve stability of each new version. The good thing that to a great extent this can be automated. Automated tests wont eliminate 100% of problems but they can deliver "good enough" results.

Another way is to signal to the consumers of updates what to expect from each new version through semantic versioning. Using version prefixes such as -alpha.0 can communicate expected instability (i.e. potential xdebug 3.0.0 incompatibility). Adding -stable communicates reliable builds (xdebug 3.0.0 in mid 2021 with a link to a migration guide). Version number itself can set expectations on what an update might include and if it is woth immediate update effort. For example a bugfix is more likely to be consumed immediately, compared to a breaking API change (major version increment). This approach provides a great flexibility in controlling the update process. But implementing it is quite hard as and it requires human judgement of version numbers. How do we know that we ONLY roll out a bug fix and not xdebug v3.0.0? It needs manual testing. Who does it? The "early adopters" (i.e. @alegatti ).

If we want to solve the problem with surpizes around xdebug v3.0.0, we need to adopt semver in image tags somwhere. For example PHP7.x-{TYPE}-{OS}-{SEMVER}. Note that semver here is the version of the docker image NOT the applications it distributes. This is somewhat similar to how helm charts in kubernetes version chart and application they distribute differently. For example: helm chart v 1.0 for pimcore can distribute pimcore 6.8. helm chart v 1.1 with a minor deployment feature added still distributes pimcore 6.8. Helm chart v 2.0 distributes pimcore 6.9, etc. Chart version increases with each build, where as application version changes less often.

The bad news is: docker does not play with semver as nicely as composer or helm do. So we need to manage semver manually by tagging and retagging docker images. i. e. php-8 == php-8.0 == php-8.0.0, updating only php-8, and php-8.0 when php-8.0.1 arrives. So even more work here.

So to put the pieces together:

• For locking we can use image digests (no work)
• For more stable image builds we need automated test suite (some work upfront, less work as it becomes mature)
• For more controll over update process we need semver (lots of work, partly manual)

This is how I see it. I hope you guys get the point. To me it seems that installing automated test suite and using digests are the best options to go forward.

[dpfaffenbauer]

Yes, I would rather add the version to the end like

# Version 1.0
pimcore/pimcore:PHP8.1-1.0.0

# Version 1.1 with minor updates of dependencies
pimcore/pimcore:PHP8.1-1.1.0
pimcore/pimcore:PHP8.1.8-1.1.0

# Version 2.0 with breaking changes or major updates of dependencies or new Base Images (eg. new debian)
pimcore/pimcore:PHP8.1-2.0.0
pimcore/pimcore:PHP8.1.8-2.0.0

# Latest Build
pimcore/pimcore:PHP8.1-latest
pimcore/pimcore:PHP8.1.8-latest

[dpfaffenbauer]

@brusch yep, fine, major and minor is enough. as well as the version prefix.

[dpfaffenbauer]

@yariksheptykin yes

[alegatti]

I looked at this repository's closed pull requests and I'm not the only one having this issue: #30 #31

I don't think that using the Pimcore major/minor version would fix this issue, because for example I could use the latest version of Pimcore but still have some other dependency in my code that is not compatible with composer 2 (this is what happened to us a few weeks ago).

Currently the docker tags have the following formats:

• PHP7.x-{TYPE} (e.g. pimcore/pimcore:PHP7.4-apache)
• PHP7.x-{TYPE}-{OS} (e.g. pimcore/pimcore:PHP7.4-apache-buster)
• the previous two formats, but with -debug appended.

What I'm proposing is to create some additional tags that contain an unique id, it could be the date in which it was built or an incremental number.
That would mean adding the following "tag formats":

• PHP7.x-{TYPE}-{BUILD_ID} (e.g. pimcore/pimcore:PHP7.4-apache-1234)
• PHP7.x-{TYPE}-{OS}-{BUILD_ID} (e.g. pimcore/pimcore:PHP7.4-apache-buster-1234)
• the previous two formats, but with -debug appended.

This would would change nothing for the people that are using the existing images (e.g. pimcore/pimcore:PHP7.4-apache), because they would be updated as usual.
But it would allow developers to choose one specific image and be sure that it will not change and break CI builds.

Of course if someone chooses to use a specific build, it's then up to them to manually check and test the new images when they are built (the same as if someone locks a minor version in composer).

[brusch]

@dpfaffenbauer also fine, but maybe we add a v prefix, to make clear it's not build metadata, but it's own version ... so e.g. pimcore/pimcore:PHP8.1-v2.0.0 ?

Also I'm wondering whether we really need the patch version part, or if major.minor would be sufficient for the sake of simplicity?

[dpfaffenbauer]

I get your problem, I also run into them from time to time (especially with the composer update).

I don't really have a nice solution for versioning a generic pimcore image honestly. Maybe we could use the Pimcore Major/Minor Releases as Tags? I don't know.... Maybe someone has a nice solution for it?

[dpfaffenbauer]

@yariksheptykin Nice summary mate :)

I'd go for something like a mixed approach here. More leaning into semver though. My latest idea was to use github (git) tags to create new version tags. That doesn't per-se limit the problem with a major software update within the image (composer 2/xdebug 3). But we could keep these dependencies hard versioned to a major version (eg. like xdebug:^2.0). We still could get version compatibilty problems with other dependencies, but not installing the latest provided version for them, isn't really a solution either. (eg. security updates)

Maybe we can get more ideas here how to solve it. I have already planned a major overhaul for these images, I don't really like having to download 1Gb per image, even if it is the same just with enabled xdebug. We still also have the permission problem to solve.

[alegatti]

I have never used digests, but I'm not sure if they would work for this case (or maybe I didn't quite understand how they work).
Let's consider this example:

• Day 1: the image pimcore/pimcore:TAG is pushed, it has the digest aaaaaa
• Day 2: I download the image, copy the digest and use it in my dockerfile.
• Day 3: a new version of pimcore/pimcore:TAG is pushed, it has the digest bbbbbb
• Day 4: Now that there is no tag pointing to aaaaaa, wouldn't the registry garbage collector delete it at some point in the future?
  To be sure that it won't be deleted shouldn't it have an unique tag?

Manifests are stored and retrieved in the registry by keying off a digest representing a hash of the contents. One of the advantages provided by CAS is security: if the contents are changed, then the digest no longer matches. This prevents any modification of the manifest by a MITM attack or an untrusted third party.
- https://docs.docker.com/registry/compatibility/#content-addressable-storage-cas

From what I understand, the digest should be used as an additional security check, to be certain that the version that I'm downloading it's the one I want and it hasn't been modified, but it's not a replacement for tags.

One strategy to deal with updates is to always use the latest beta untested version. [...] We can of course let this happen only in safe environments such as local dev env.
- @yariksheptykin

I don't share this view. IMHO the dev environment should be locked just like the production ones. It's up to the developers to check the new versions when they are released. I don't want my dev env to break unexpectedly while I'm working on something.

Yeah, having semver for the images would be great, but it requires a lot more work.
My proposed solution was less flexible but simpler: the new images always have the latest stable versions (that are compatible with pimcore) of the several dependencies needed (composer, xdebug, wkhtmltopdf, etc...);.
The developer can rest assured that the images do not change without warning if they use -{BUILD_ID}, and they have time to update their code to work with the new image.
Older versions don't receive updates (not even security ones).

Anyway, whatever solution we choose to implement, I think that (at least in the short term) having the dependencies limited to the major versions is a great idea.

@yariksheptykin What kind of automated tests were you thinking of implementing for this image? I'm usually completely on board with the suggestion of adding tests, but I don't really see what could be done here: most of the incompatibility problems will arise from something to do with my code, not in the pimcore image itself since it's almost just a php image with a bunch of different dependencies installed on top.

@dpfaffenbauer do you have any idea on how to refactor the images to reduce the size? Should this be discussed in another issue?

[brusch]

What do you guys think about the following approach:

We introduce release branches in this repo, this allows us to maintain multiple versions of our images without having a mess of if/else/.... So for example we start with v1 and we try to pin versions of packages wherever it is possible or makes sense, e.g. xdebug or ImageMagick. Then we also have a dev or latest branch, which will be similar to the current main branch. We can then also schedule the jobs for the release branches to get the latest security updates and so on, but without breaking anything.

The tag would then look like the following examples:

pimcore/pimcore:v1-PHP8.1
pimcore/pimcore:dev-PHP8.1-debug
pimcore/pimcore:v1-PHP8.1-supervisor

pimcore/pimcore:PHP8.1 # always points to the latest version, e.g. v4
# or
pimcore/pimcore:latest-PHP8.1

I wouldn't use fpm in the tag anymore, as we will drop the cli and then it could be misleading.
We also don't need the distro e.g. ´bullseyeanymore, because this will be covered byv1` - no distro change within a version!

We should also think about adding additional tags for the full PHP version, because there were already situations where PHP had some major problems and you had to revert to an older version.

# main tag
pimcore/pimcore:v1-PHP8.1

# specific tag
pimcore/pimcore:v1-PHP8.1.8

We could automate that by getting the version number out of the latest image:
docker run -it --rm php:8.1-fpm-bullseye php -v

WDYT? @dpfaffenbauer @alegatti @yariksheptykin

[brusch]

So pimcore/pimcore:PHP8.1-v2.0 it is 😊

More examples:

pimcore/pimcore:PHP8.1-latest
pimcore/pimcore:PHP8.1-dev
pimcore/pimcore:PHP8.1-supervisor-v1.0
pimcore/pimcore:PHP8.1-debug-latest

[yariksheptykin]

@brusch that -v2.0 corresponds to a release v2.0 in pimcore/docker repo, right? So this is basically an immutable tag. New changes accumulate in -dev flavor, and, when mature enough, get released / locked with "-vX.X" immutable tags. Do I understand it right?

[maxjuergens]

So pimcore/pimcore:PHP8.1-v2.0 it is blush

More examples:

pimcore/pimcore:PHP8.1-latest
pimcore/pimcore:PHP8.1-dev
pimcore/pimcore:PHP8.1-supervisor-v1.0
pimcore/pimcore:PHP8.1-debug-latest

I'd go the official Docker images naming route, and the latest suggestion is nearly there:

• Tags should be lower case
• latest Tag should point to the latest version/release (pimcore/pimcore:latest, not pimcore/pimcore:PHP8.1-latest)

Examples:

pimcore/pimcore:latest (just the latest and greatest)
pimcore/pimcore:php8.1 (less specific variant)
pimcore/pimcore:php8.1.8-v1.0.0 (very specific (immutable) variant)

pimcore/pimcore:php8.1-supervisor
pimcore/pimcore:php8.1.8-supervisor-v1.0.0

pimcore/pimcore:php8.1-debug (or with -xdebug, allows to set Xdebug versions if needed, e.g. pimcore/pimcore:php8.1-xdebug3.1.5)
pimcore/pimcore:php8.1.8-debug-v1.0.0

And maybe someday (this should be a goal imho):

pimcore/pimcore:latest
pimcore/pimcore:10-php8 (the 10 is the Pimcore version, since it's bundled with the Docker image)
pimcore/pimcore:10.5-php8.1
pimcore/pimcore:10.5.0-php8.1.8-v1.0.0

Take a look at e.g. Drupal image tags for a reference.

[dpfaffenbauer]

And maybe someday (this should be a goal imho):

pimcore/pimcore:10-php8 (the 10 is the Pimcore version, since it's bundled with the Docker image)
pimcore/pimcore:10.5-php8.1
pimcore/pimcore:10.5.0-php8.1.8-v1.0.0```

I disagree with you here. If you bundle Pimcore with it, you cannot change code without installing it outside the image anyway. This is image is supposed to be a base-image for development and running Pimcore containerized.

If you need your code bundled into the image, that is up to you.

Re Versions: lower-case tags also fine with me. The rest is up to @brusch ?

[brusch]

@dpfaffenbauer lowercase fine 4 me 😉
@maxjuergens

you basically have a PHP variant and not Pimcore

Not really because it also includes all Pimcore dependencies, which are not only limited to PHP (such as LibreOffice, ...)

[brusch]

First release done, so resolved, see also readme 🤗