tiangolo / docker-with-compose

Docker image with Docker Compose installed for CI.

🚨 DEPRECATION WARNING 🚨

As Docker now has compose built in there's no longer need for this Docker image.

You should use the official Docker image instead.

---

Supported tags and respective Dockerfile links

• latest (Dockerfile)

Note: There are tags for each build date. If you need to "pin" the Docker image version you use, you can select one of those tags. E.g. tiangolo/docker-with-compose:2021-09-17.

Docker with Docker Compose image

Docker image with Docker Compose installed for CI.

Description

The main purpose of this image is to help in Continuous Integration environments that need the docker binary, the docker-compose binary, and possibly require doing other small things, like running shell scripts or notifying some API with curl.

It includes both programs (docker and docker-compose) and allows to run arbitrary shell scripts (contrary to the official Docker Compose image).

By not having to install docker-compose on top of a docker:latest image it can reduce the building time about 10 / 15 seconds in a cloud data center for each build. In environments in where the Internet connection is less good than a cloud provider, the time saved would be more.

GitHub repo: https://github.com/tiangolo/docker-with-compose

Docker Hub image: https://hub.docker.com/r/tiangolo/docker-with-compose/

Usage

Pull the image:

$ docker pull tiangolo/docker-with-compose

Then run a container of this image mounting the Docker sock as a host volume.

By mounting the Docker sock as a volume you allow the docker client inside of the container to communicate with your Docker (the Docker daemon/service) on your machine directly.

This way, you can send Docker commands, like pulling, running, or building a new Docker image, from inside this container.

You might also want to mount a host volume with the files that you need to use.

---

For example, let's say you have a Dockerfile like:

FROM nginx

RUN echo "Hello World" > /usr/share/nginx/html/index.html

You could:

• Mount the local directory containing that Dockerfile.
• Mount the local Docker sock.
• Build that Nginx image from inside of container running this image.

$ docker run -v $(pwd):/app -v /var/run/docker.sock:/var/run/docker.sock tiangolo/docker-with-compose sh -c "cd /app/ && docker build -t custom-nginx ."

Problem description

There is an official Docker image that contains the docker binary. And there is a Docker Compose image.

But the Docker Compose image has docker-compose as the entrypoint.

So, it's not possible to run other commands on that image, like installing something, e.g. apt-get install -y curl.

And it's also not possible to run docker commands directly, e.g. docker login -u ci-user -p $CI_JOB_TOKEN $CI_REGISTRY.

This image allows running arbitrary commands like shell scripts, docker commands and also Docker Compose commands like docker-compose build and docker-compose push.

As several Continuous Integration systems allow doing previous steps, like installing packages before running the actual main script, those steps could be used to install Docker Compose. But by downloading and installing Docker Compose every time, the builds would be slower.

For example, a very simple GitLab CI file .gitlab-ci.yml could look like:

# Do not use this file example
image: docker:latest

before_script:
  - apk add --no-cache py-pip
  - pip install docker-compose
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

ci:
  script:
    - docker-compose build
    - docker-compose up -d
    - docker-compose exec -T tests run-tests.sh
    - docker-compose down -v
    - docker stack deploy -c docker-compose.prod.yml --with-registry-auth prod-example-com

But when the base image has to download and install Docker Compose every time, that's time added to the process. Specifically the lines in:

...

  - apk add --no-cache py-pip
  - pip install docker-compose

...

This image's solution

This image includes Docker Compose and allows you to run any other arbitrary command.

So your GitLab CI .gitlab-ci.yml file could then look like:

image: tiangolo/docker-with-compose

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

ci:
  script:
    - docker-compose build
    - docker-compose up -d
    - docker-compose exec -T tests run-tests.sh
    - docker-compose down -v
    - docker stack deploy -c docker-compose.prod.yml --with-registry-auth prod-example-com

And it would run faster as it doesn't have to install Docker Compose every time.

And you could start that initial GitLab Runner following the GitLab CI runner for CI/CD guide on DockerSwarm.rocks.

The same would apply for Travis, Jenkins or whichever CI system you use.

Release Notes

Latest Changes

• 🗑️ Deprecate this project, use official Docker instead 🎉. PR #47 by @tiangolo.
• ⬆️ Bump actions/checkout from 3.1.0 to 3.3.0. PR #44 by @dependabot[bot].
• 👷 Update Latest Changes token. PR #46 by @tiangolo.
• 👷 Add GitHub Action for Docker Hub description. PR #41 by @tiangolo.
• ⬆️ Upgrade CI OS. PR #40 by @tiangolo.
• 🔧 Add funding config. PR #39 by @tiangolo.
• 👷 Add automatic scheduled CI every monday. PR #38 by @tiangolo.
• 👷 Add automatic scheduled CI every Monday. PR #37 by @tiangolo.
• 📝 Update README, replace bash with shell, as Bash itself is not installed. PR #36 by @tiangolo.
• 👷 Add alls-green GitHub Action. PR #35 by @tiangolo.
• 👷 Do not run double CI for PRs, run on push only on master. PR #34 by @tiangolo.
• ⬆️ Bump tiangolo/issue-manager from 0.3.0 to 0.4.0. PR #28 by @dependabot[bot].
• Bump actions/checkout from 2 to 3.1.0. PR #31 by @dependabot[bot].
• 🐛 Fix deployment. PR #26 by @tiangolo.
• 🐛 Fix GitHub Actions and latest requirements. PR #25 by @tiangolo.
• 👷 Move from Travis to GitHub Actions. PR #23 by @tiangolo.
• ✨ Add external dependencies and Dependabot to get automated upgrade PRs. PR #27 by @tiangolo.
• 👷 Add Latest Changes GitHub Action. PR #24 by @tiangolo.
• Upgrade Python to use version 3.x. PR #15.
• Add curl to the installed and available packages. PR #14 by @stratosgear.
• Add Travis CI. PR #4.
• Upgrade Docker Compose installation. PR #3 by @boskiv.

License

This project is licensed under the terms of the MIT license.

docker-with-compose's Issues

proposal: include `gettext`

gettext allows me to use envsubst which is nice for configuring rendering docker-compose.yml with external secrets, networks etc as their names better to be stored as CI variables rather as hardcoded value in docker-compose.yml.

proposal: include `docker-auto-labels`

In full-stack-fastapi-postgresql cookiecutter repo docker-auto-labels were used, and I think that including this package could speedup each CI pipeline a bit.

Running docker-compose on python3 would work better

docker-compose has some annoying bugs running on python2, e.g. not seeing log output of services that have been idle for some time

Furthermore, docker-compose is considering deprecating using it on Python 2 altogether.

We should use a base image with Python 3 and run docker-compose on that instead of Python 2.

GitLab CI/CD - Couldn't connect to docker daemon

Hi everyone
I just tried to use this useful image but getting error trying a simple docker-compose build ?

.gitlab-ci.ymlseems to me like copy-pasted from official docs, not sure I'm missing something? pretty new to GitLab builds, so I could miss something obvious.

Output from CI:

thanks
nicola

Docker-compose now requires Alpine cargo package

The Dockerfile fails to build now because apt-get install cargo is required for docker-compose. See:

https://docs.docker.com/compose/install/#alternative-install-options

update docker-compose to 1.25

add make

Currently, I have to install make everytime I want to run make commands. IMO, a good amount of people may be using make to run docker-compose commands.

Is it out of the scope to add the make installation to this package?

This is my current .gitlab-ci.yml file

image: tiangolo/docker-with-compose

services:
  - docker:dind

before_script:
  - apk add --update make

stages:
  - lint
  - test

lint:
  stage: lint
  script:
    - make ci-lint

test:
  stage: test
  script:
    - make ci-test

somehow not working anymore with gitlab

I had your image (thanks very much) set up with a private gitlab instance, but it just stopped working. Though there are no obvious changes on the server where it is executed nor in your repo. But maybe you have some idea?
config.toml and erros message below.

Running with gitlab-runner 11.11.2 (ac2a293c)
  on homeserver something
Using Docker executor with image tiangolo/docker-with-compose ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:6ce0d31cf4d670f84b0f74b1b15910f9e3fde4105232e14b0d360a6a49c05362 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image tiangolo/docker-with-compose ...
Using docker image sha256:cf0fa05056110e8c9866200f6879b615ea55be3e54f558826aab5b0882537f70 for tiangolo/docker-with-compose ...
Running on runner-something-0 via ubuntu...
DEPRECATION: this GitLab server doesn't support refspecs, gitlab-runner 12.0 will no longer work with this version of GitLab
Fetching changes...
Checking out 13232142 as somebranch...

Skipping Git submodules setup
$ docker-compose up -d
Couldn't connect to Docker daemon at http://docker:2375 - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
ERROR: Job failed: exit code 1

config.toml

concurrent = 1
check_interval = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "homeserver"
url = "https://gitlab.something.eu/"
token = "<sometoken>"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "tiangolo/docker-with-compose"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
  [runners.cache.s3]
  [runners.cache.gcs]

Bitbucket Pipelines: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

First of all, thanks a lot for this project.

Just wanted to let you (and others) know that the latest version (latest / 2022-11-10) doesn't work in Bitbucket Pipelines anymore:

+ docker-compose build
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1
#1 pulling image moby/buildkit:buildx-stable-1 2.6s done
#1 creating container buildx_buildkit_default 0.0s done
#1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed
------
 > [internal] booting buildkit:
------
Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

I switched back to version 2021-09-18, which works fine.
Not sure if this is anything you can fix at all.

Time ot sunset this repo?

Now that compose has been added into the official docker image (docker-library/docker#361) what is the purpose of this repository anymore?

Is it time to tell folks to stop using it? Should it become a plain mirror for docker:latest?

Can't access compose container port on localhost in Gitlab CI

Hi,

I have a docker-compose.yaml that runs the Keycloak SSO server and Postgresql. The Keycloak HTTP server listens on port 8080 and so I have

-ports
  8080:8080

in the docker-compose.yaml.

In my .gitlab-ci.yml I do: docker-compose -f path/to/docker.compose.yaml up -d. I then start a watchdog so that I know when Keycloak is available by doing a curl -I http://localhost:8080 in a loop-with-sleep checking curl's exit status.

However, I get connection refused even when I know the Keycloak is up and running. I'm sure there's something I don't understand here. What is it?

CURL

can you add curl installation? may with same tag on docker image

FROM docker:latest

RUN apk add --no-cache curl
RUN apk add --no-cache py-pip
RUN pip install docker-compose

Stack generation have breaking changes from 2021-09-18 to latest

The latest push did break some of our deployments.

Initially posted in #32 but does not per se have the same issue.

Before:

FROM tiangolo/docker-with-compose:2021-09-18
WORKDIR /tmp
COPY docker-compose.yml ./
CMD ['docker-compose', '-f', 'docker-compose.yml', 'config']

After:

FROM tiangolo/docker-with-compose:latest
WORKDIR /tmp
COPY docker-compose.yml ./
CMD ['docker-compose', '-f', 'docker-compose.yml', 'config']

Basic file input docker-compose.yml

version: '3.8'
 
services:
  db:
    image: postgres:14-alpine
    environment:
      POSTGRES_USER: api
      POSTGRES_PASSWORD: azerty
      POSTGRES_DB: api
    volumes:
      - app-db:/var/lib/postgresql/data
 
volumes:
  app-db:

Output before:

services:
  db:
    environment:
      POSTGRES_DB: api
      POSTGRES_PASSWORD: azerty
      POSTGRES_USER: api
    image: postgres:14-alpine
    volumes:
    - app-db:/var/lib/postgresql/data:rw
version: '3.8'
volumes:
  app-db: {}

Output now:

name: tmp
services:
  db:
    environment:
      POSTGRES_DB: api
      POSTGRES_PASSWORD: azerty
      POSTGRES_USER: api
    image: postgres:14-alpine
    networks:
      default: null
    volumes:
    - type: volume
      source: app-db
      target: /var/lib/postgresql/data
      volume: {}
networks:
  default:
    name: tmp_default
volumes:
  app-db:
    name: tmp_app-db

• version: '3.8' is not present on the generated stack.yml (the target host fallback to version: 1 which is problematic. even though it is on our (input) files.
• volumes descriptions have changed unexpectedly so there are re-created (not found)
• it seems that this image generates a compose-compliant (v2) schema file while the command docker stack is not ready.

from remote, not actually the docker-with-compose image

docker stack deploy -c docker-stack.yml --with-registry-auth "xxxx"
unsupported Compose file version: 1.0

hint: the docker-stack.yml seems to be generated using compose v2 instead of docker-compose legacy version.

Different dependencies installed with apk

Running apk to install nodejs no longer installed npm depdendency.

Note that nghttp2-libs is installed, and nodejs-npm is not.

Left: Using current image (tiangolo/docker-with-compose:latest)
Right: Using image from a day ago

According to the docker-with-compose build details, the build was just updated.

Different versions of alpine are noted in the fetch comments.

/usr/local/bin/docker-entrypoint.sh: exec: line 61: /bin/bash: not found

/usr/local/bin/docker-entrypoint.sh: exec: line 61: /bin/bash: not found

New version of Docker Compose: 1.26.2

When I run docker-compose -version on my Mac I get:

1.26.2, build eefe0d31

Inside the tiangolo/docker-with-compose container I get:

1.26.0

I have an issue using contexts that only repros in the container, and I assume it's from the version difference. When I built the container manually, i.e. ran docker:latest then ran apk add --no-cache py3-pip python3-dev libffi-dev openssl-dev curl gcc libc-dev make && pip3 install docker-compose my issue disappeared.

Can you rebuild your docker image, to get the newest docker-compose included?

Bash

The README says:

This image allows running arbitrary commands like Bash scripts,

But the image does not contain bash. I propose we apk add bash.

docker-compose up -d fails

Error -

Couldn't connect to Docker daemon at http+docker://localhost - is it running?

If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

[FEATURE] Document how to mount the Docker sock

Document how to mount the Docker sock as a volume so that Docker Compose inside the container can have access to the local Docker service/daemon.

This has to be done instead of Docker-in-Docker: http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/

Probably with the ideas here: https://medium.com/better-programming/about-var-run-docker-sock-3bfd276e12fd

Typo in the README.md file

Inside the Description section of the README.md file, there is a typo:

The main purpose of this image is to help in Continuous Integration environments that need the docker binary, the docker-compose binary and posibly (possibly) require doing other things, like running Bash scripts.

Not working with gitlab

This image wasn't working for me in gitlab. (I'm running tests for the fastAPI project generator library) I was using the following:

image: tiangolo/docker-with-compose

before_script:
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  - docker info
  - pip install docker-auto-labels

variables:
  DOCKER_TLS_CERTDIR: ""

stages:
  - test

tests:
  stage: test
  script:
    - sh ./scripts/test.sh

But would get the following error: error during connect: Post http://docker:2375/v1.40/auth: dial tcp: lookup docker on 169.254.169.254:53: no such host (see the job here)

Any ideas why this image isn't working? Is it due to the TLS changes with docker:dind?

Using the official docker/compose image, and changing the entrypoint, is working for me:

image:
  name: docker/compose:1.25.1-rc1
  entrypoint: ["/bin/sh", "-c"]

variables:
  DOCKER_TLS_CERTDIR: ""
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2

services:
  - docker:dind

before_script:
  - docker info
  - docker-compose version

stages:
  - test

tests:
  stage: test
  script:
    - sh ./scripts/test.sh