GithubHelp home page GithubHelp logo

smartrecruiters / docker-bakery Goto Github PK

View Code? Open in Web Editor NEW
41.0 26.0 2.0 30.63 MB

Dockerfile hierarchy management tool with automatic rebuilding of dependent images when parent changes

License: MIT License

Makefile 0.91% Go 98.72% Shell 0.38%
docker bakery images docker-hierarchy dockerfile-goodness dockerfile-generator dockerfiles dockerfiles-management builder dockerfiles-tree

docker-bakery's Introduction

docker-bakery

Build Software License Go Report Card Go Doc

Purpose

Aim of the docker-bakery is to provide simple solution for automatic rebuilding of dependent images when parent image changes.

Features

  • Automatic triggering of dependant images builds when parent changes
  • Support for Dockerfile templating with usage of golang template engine
  • Support for semantic versioning scope changes
  • Possibility to build and push docker images to custom registries
  • Possibility of providing custom build and push commands
  • Versioning with git tags
  • Written in golang

Example usage

"Example usage"

Example project

See docker-bakery-example to check how this tool works in action.

Assumptions

  • convention: image name is equal to the parent directory name

  • presence of Dockerfile.template will qualify image for automatic updates triggered by base images (FROM clause is analyzed)

  • first line in the dockerfile must start with FROM otherwise dependency will not be determined for that file

  • scope change in the base image is propagated to the child images

  • to benefit from dependency updates the child image must use in template variable according to the convention: {{.BASE_IMAGE_NAME_VERSION}} where BASE_IMAGE_NAME should be substituted with uppercase directory name of the base image.

    Any hyphens (-) or dots (.) should be replaced with the underscores (_) in the variable name.

    For example the correct variable name for image/directory named jdk8-gradle2.14 is {{.JDK8_GRADLE2_14_VERSION}}

  • docker file templates need to be placed in git repository (with defined remote) in order versioning of the images could work (versioning is done via git tags)

  • additional dynamic variables will be accessible for build templating

Config

docker-bakery needs config.json to be provided. Config is used for templating Dockerfile.template files and build commands Structure of the config.json file is as follows:

 {
	"properties": {
		"DEFAULT_PULL_REGISTRY": "some-private-registry.com:9084",
		"DEFAULT_PUSH_REGISTRY": "some-private-registry.com:9082",
		"DOCKERFILE_DIR": "Reserved dynamic property, contain path to currently build image. Can be used in template.",
		"IMAGE_NAME": "Reserved dynamic property, represents image name. Can be used in template.",
		"IMAGE_VERSION": "Reserved dynamic property, represents new version of the image. Can be used in template."
	},
	"commands": {
		"defaultBuildCommand": "docker build --tag {{.IMAGE_NAME}}:{{.IMAGE_VERSION}} --tag {{.DEFAULT_PUSH_REGISTRY}}/{{.IMAGE_NAME}}:{{.IMAGE_VERSION}} --tag {{.DEFAULT_PULL_REGISTRY}}/{{.IMAGE_NAME}}:{{.IMAGE_VERSION}} {{.DOCKERFILE_DIR}}",
		"defaultPushCommand": "docker push {{.DEFAULT_PUSH_REGISTRY}}/{{.IMAGE_NAME}}:{{.IMAGE_VERSION}}"
	},
	"reportFileName": "custom-report-filename.json",
	"verbose": false,
	"autoBuildExcludes": [
		"some-image-name-that-will-be-excluded-from-build-when-parent-changes"
	]
 }

Properties config section

This section is dedicated for storing any custom properties that may be available for usage in Dockerfile.template files. Feel free to modify this section and provide properties according to your needs. Flat structure should be preserved.

This section will also be updated with dynamic properties during runtime. Dynamic properties do not have to be defined in config as they are automatically added during runtime.

Following properties belong to dynamic ones:

  • DOCKERFILE_DIR - will be replaced with currently processed dockerfile dir
  • IMAGE_NAME - will be replaced with currently processed image name
  • IMAGE_VERSION - will be replaced with currently processed image version
  • *_VERSION - where * is the image name. There will be that many properties of this kind as many images are in hierarchy. Initially those properties will be filled with latest versions of pushed images.
  • BAKERY_BUILDER_NAME - will be replaced with the git user name (taken from git config user.name)
  • BAKERY_BUILDER_EMAIL - will be replaced with the git user email (taken from git config user.email)
  • BAKERY_BUILDER_HOST - will be replaced with hostname of the machine where build is executed
  • BAKERY_BUILD_DATE - will be replaced with current build date
  • BAKERY_IMAGE_HIERARCHY - will be replaced with the path representing image hierarchy in the following format:

parent1:versionOfParent1->parent2:versionOfParent2->imageName:imageVersion

Hierarchy is built automatically given that parent images are exporting the same ENV variable that can be accessed in child images. Check the example project for references.

  • BAKERY_SIGNATURE_VALUE - will be replaced with a one liner string value embedding other BAKERY* variables together. Can be used in templates to create for example ENV variable. Example:

SINGATURE=Builder Name;[email protected];builder-host-name;2018-03-16 15:47:58;alpine-java:8u144b01_jdk->mammal:3.2.0->dog:4.0.0->dobermann:4.0.0->smaller-dobermann:4.0.0

  • BAKERY_SIGNATURE_ENVS - will be replaced with embedded BAKERY* variables in a key=value format. Convenient if you wish to have all BAKERY* variables in a dockerfile under single key. Check the example project for references.

Commands config section

This section contains two templates used for building and pushing docker images. It allows for specifying custom parameters. Commands defined here as templates will be filled with available defined properties from the config section + the dynamic properties set during runtime.

Other config attributes

reportFileName - if set it will be used as a file name to store information (in JSON format) about successfully built images.

Dockerfile.template

Presence of the Dockerfile.template file qualifies the image for the place in hierarchy and therefore allows for triggering builds that depend from this image. It also ensures that image build will be triggered when its parent changes.

Usage

To make use of docker-bakery as convenient as possible checkout usage of Makefiles from the example project that will simplify usage greatly. If you don't want to use makefiles you can still use docker-bakery tool directly. Checkout the CLI help via docker-bakery -h.

Command help

COMMANDS:
     fill-template, prepare, prepare-recipe         Used to fill Dockerfile.template file. Values needed for template are taken from the config file and from dynamic properties provided during runtime.
     build                                          Used to build next version of the images in given scope. Optionally it can skip build of dependant images.
     push                                           Used to push next version of the images in given scope. Optionally it can skip push of dependant images.
     show-structure, ss, show-hierarchy, hierarchy  Used to display hierarchy of the images
     dump-latest-versions, dump                     Used to dump data about latest versions of images to the provided file
     help, h                                        Shows a list of commands or help for one command

Command fill-template

docker-bakery fill-template -h
NAME:
   docker-bakery fill-template - Used to fill Dockerfile.template file. Values needed for template are taken from the config file and from dynamic properties provided during runtime.

USAGE:
   docker-bakery fill-template [command options] [arguments...]

OPTIONS:
   --input value, -i value      Required. Input Dockerfile.template.
   --output value, -o value     Required. Output Dockerfile generated from template.
   --config value, -c value     Required. Path to config.json with properties and build commands defined.
   --rootDir value, --rd value  Optional. Used to override rootDir of the dockerfiles location. Can be defined in config.json, provided in this argument or determined dynamically from the base dir of config file.

Command build

docker-bakery build -h
NAME:
   docker-bakery build - Used to build next version of the images in given scope. Optionally it can skip build of dependant images.

USAGE:
   docker-bakery build [command options] [arguments...]

OPTIONS:
   --dockerfile value, -d value  Required. Path to dockerfile/dockerfile.template file that needs to be build.
   --scope value, -s value       Required. Scope of the change used to generate the next version. Can be one of: major/minor/patch.
   --config value, -c value      Required. Path to config.json with properties and build commands defined.
   --root-dir value, --rd value  Optional. Used to override rootDir of the dockerfiles location. Can be defined in config.json, provided in this argument or determined dynamically from the base dir of config file.
   --skip-dependants, --sd       Optional. False be default. If this flag is set build of the parent will not trigger dependant builds.
   --property value, -p value    Optional. Allows for providing additional multiple properties that can be used during templating. Overrides properties defined in config.json file. Expected format is: -p propertyName=propertyValue
     

Command push

docker-bakery push -h
NAME:
   docker-bakery push - Used to push next version of the images in given scope. Optionally it can skip push of dependant images.

USAGE:
   docker-bakery push [command options] [arguments...]

OPTIONS:
   --dockerfile value, -d value  Required. Path to the dockerfile/dockerfile.template that needs to be pushed.
   --scope value, -s value       Required. Scope of the change used to generate the next version. Can be one of: major/minor/patch.
   --config value, -c value      Required. Path to config.json with properties and build commands defined.
   --rootDir value, --rd value   Optional. Used to override rootDir of the dockerfiles location. Can be defined in config.json, provided in this argument or determined dynamically from the base dir of config file.
   --skip-dependants, --sd       Optional. False be default. If this flag is set build of the parent will not trigger dependant builds.

Command copy-images-hierarchy

Let's say you have a given hierarchy of images:

python-3.8
    python-3.8-jdk
    python-3.8-web
        python-3.8-flask

If you want to quickly introduce python3.9 for those images, you can:

  1. create python-3.9 image
  2. recreate images hierarchy
docker-bakery copy-images-hierarchy --config ${path_to_config} --base-image sr-python3.8 -rp 3.8=3.9 -r   

The resulting hierarchy will look like this:

python-3.9
    python-3.9-jdk
    python-3.9-web
        python-3.9-flask

For more options run docker-bakery copy-images-hierarchy help

How to apply it to your project

Applying docker-bakery is quite simple. Take a look here

Limitations

At the moment docker multi-stage builds are not fully supported. Only the first line FROM is taken into consideration when determining the parent of the image and its place in the hierarchy.

docker-bakery's People

Contributors

gmiejski avatar jaroslaw-bochniak avatar kasinskim avatar kszarek avatar t-lesniewski-sr avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docker-bakery's Issues

List all images and build only the ones that changed

Hi!

I am trying to use docker-bakery in my CI/CD pipelines. The goal is to, whenever someone merges to master, build only the images that changed, and their children, avoiding building the children twice if both the parent and the children changed.

This is easy locally, because you know which files you edited, so you just go to those and run the build command. In the CI/CD (I'm using Gitlab), I can list all the Docker.template files that changed since the last merge, but I still need to find a way to avoid building parent and child if both changed. My idea was to use the show-structure command and filter using that, but the output is very difficult to parse. I think adding a json output option would be very useful.

I also tried using the images' digests to detect if something changed or not, but those change every time, even if the image is exactly the same, I think this happens because of the BAKERY_BUILD_DATE env variable that changes every time automatically.

I am also open to suggestions as to how automate this. Maybe I am just doing everything wrong! ๐Ÿ˜„

Thank you in advance!

Custom property in image name breaks hierarchy determination

Hey,

for my use case I'm including the current branch name within the generated image name. So I use something like FROM {{.DEFAULT_PULL_REGISTRY}}/base/{{.BRANCH}}:{{.BASE_VERSION}} in child images (Full minimal testcase). When determining the hierarchy, this property does not seem to be evaluated properly:

$ docker-bakery build -s major -c config.json -d base/Dockerfile.template -p BRANCH=master

Dockerfiles hierarchy discovered in .
โ”œโ”€โ”€ scratch
โ”‚   โ””โ”€โ”€ base
โ””โ”€โ”€ {{.BRANCH}}
    โ””โ”€โ”€ child

As a result, only base gets build, because child is not being determined as a child of base.

Image names containing hierarchy

We are interested in using this tool for our internal use case. However, we found one thing that we would like to have in a different way. Currently, the folder name where the Dockerfile.template is located is used as the IMAGE_NAME. Let me explain it using your example repo. You have the following structure:

โ”œโ”€โ”€ alpine-java
โ”‚   โ”œโ”€โ”€ bird (latest: 0.0.1)
โ”‚   โ””โ”€โ”€ mammal (latest: 3.2.5)
โ”‚       โ”œโ”€โ”€ cat (latest: 5.0.4)
โ”‚       โ”œโ”€โ”€ dog (latest: 5.0.4)
โ”‚       โ”‚   โ”œโ”€โ”€ dobermann (latest: 4.0.4)
โ”‚       โ”‚   โ”‚   โ””โ”€โ”€ smaller-dobermann (latest: 4.0.4)
โ”‚       โ”‚   โ”œโ”€โ”€ pitbull (latest: 3.0.4)
โ”‚       โ”‚   โ””โ”€โ”€ labrador (latest: 9.0.4)
โ”‚       โ””โ”€โ”€ horse (latest: 3.2.4)
โ””โ”€โ”€ scratch
    โ””โ”€โ”€ fish (latest: 1.0.0)

Right now all the images have the following name structure: {{.DEFAULT_PULL_REGISTRY}}/<name>:<version>. However, we want a nested structure, an example for dobermann would be: {{.DEFAULT_PULL_REGISTRY}}/mammal/dog/dobermann:<version>. I think this solution is more verbose than having to define folder names like mammal-dog-dobermann to achieve something similar. As well, if 2 directories have the same name but different parents, the current approach would cause issues.

Is this something you could add support to? Or is this project not maintained anymore?

If this is something you would accept I can create a pull request for it

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.