running docker command hangs about packer-templates

[kwilczynski]

@eli007 thank you for letting me know. I will have a look. Although, I have to admit that I am using this in production and never seen this happen before. Perhaps something has changed with the latest version of Docker. I will rebuild the AMI from scratch and get back to you.

[kwilczynski]

@eli007 I have built a fresh image and brought it up. Then I could see the following:

root 796 1 0 09:41 ? 00:00:00 /usr/bin/docker daemon -H fd://

Docker daemon running.

ubuntu@ip-10-0-0-134:~$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-8:1-399196-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 9.411 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.110 (2015-10-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.0-28-generic
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 990.8 MiB
Name: ip-10-0-0-134
ID: NNLP:B7DC:AAHR:DDPP:WLT2:PUER:4C3U:6IFU:PQYE:IAUJ:SXWL:PKHU
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

The warning aside, which needs looking into at some point (some changes by Docker).

Trying some commands...

ubuntu@ip-10-0-0-134:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
ubuntu@ip-10-0-0-134:~$ docker ps -a
CONTAINER ID        IMAGE

Nothing hangs and/or freezes.

Then, running some containers:

ubuntu@ip-10-0-0-134:~$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

ubuntu@ip-10-0-0-134:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
1368317e940f        hello-world         "/hello"            25 seconds ago      Exited (0) 19 seconds ago                       lonely_goldstine
ubuntu@ip-10-0-0-134:~$ docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
90d6565b970a: Pull complete
40553bdb8474: Pull complete
c3129e7479ab: Pull complete
091663bd70db: Pull complete
Digest: sha256:ba1688fec34f66d8a7ff5b42e6971625d8232c72bf0e38ad06dda10cad157293
Status: Downloaded newer image for ubuntu:latest
root@bc89824365a4:/# date
Tue Jul 12 09:48:25 UTC 2016
root@bc89824365a4:/# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 09:48 ?        00:00:00 bash
root        10     1  0 09:48 ?        00:00:00 ps -ef
root@bc89824365a4:/# whoami
root
root@bc89824365a4:/# exit

Everything seem in order.

Can you try to reproduce this using the "hello-world" and the "ubuntu" upstream images? I want to make sure that it's not the way how everything was installed, which is really very basic.

[eli007s]

can you provide an example of how you are building the packer build with the env being passed in?

Perhaps the difference could be the versions being used. I am not passing in any version numbers for docker-compose ordocker.

I'll try a fresh build with a new AMI again to see if maybe it was the AMI itself causing the issues.

[kwilczynski]

@eli007 I have a build process setup which renders a variables.json file as follows (example):

{
  "version": "0.1.5",
  "account_id": "",
  "access_key": "",
  "secret_key": "",
  "region": "eu-west-1",
  "vpc_id": "vpc-12345678",
  "subnet_id": "subnet-12345678",
  "instance_type": "m3.medium",
  "source_ami": "",
  "headless": "false"
}

Then it runs it like this (example): packer build -var-file variables.json -only amazon-ebs ubuntu-16.04-amd64-docker.json

The source_ami was ami-a4d44ed7, which is SSD EBS-backed and HVM, and these can be obtained from: https://cloud-images.ubuntu.com/locator/ec2/

I have a small VPC built with Terraform that sets the bare minimal network for Packer to work.

To add, I have also not set any specific version for any of the software packages (e.g. Docker, Docker Compose, etc.) to be installed.

[kwilczynski]

@eli007 how did your investigation go?

[eli007s]

I think my problem might have been a t2.micro =)

might bee too small of an instance to handle docker

[kwilczynski]

@eli007 that might be an issue if your test container runs things that are more demanding, as you only have 1 GB of RAM to share on t2.micro. The "Hello World" example should definitely work, though.

Did you have more luck running containers on different instance type? You can always build Vagrant box if you need to experiment first. Saves you money on an Amazon bill.

[kwilczynski]

@eli007 I hope that you don't have any more issues. I am going to close this issue, but re-open if needed.