streamnsight / docker-flink

Apache Flink cluster deployment in Docker containers using Docker-Compose

#Apache Flink cluster deployment on Docker using Docker-Compose

##Installation ###Install Docker

https://docs.docker.com/installation/

if you have issues with Docker-Compose versions incompatible with your version of Docker try

curl -sSL https://get.docker.com/ubuntu/ | sudo sh

###Install Docker-Compose

curl -L https://github.com/docker/compose/releases/download/1.1.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose

chmod +x /usr/local/bin/docker-compose

###Get the repo

###Build the images

Images are based on Ubuntu Trusty 14.04 and run Supervor to stay alive when running containers.

The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.

• Run ./build.sh

###Deploy

• Deploy cluster and see config/setup log output (best run in a screen session)

docker-compose up

• Deploy as a daemon (and return)

docker-compose up -d

• Scale the cluster up or down to N TaskManagers

docker-compose scale taskmanager=<N>

• Access the JobManager node with SSH (exposed on Port 220)

ssh root@localhost -p 220

or on Mac OS X with boot2docker

ssh root@$(boot2docker ip) -p 220

The password is 'secret'

• Kill the cluster

docker-compose kill

• Upload a jar to the cluster

scp -P 220 <your_jar> root@localhost:/<your_path>

• Run a topology

ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>

or

ssh to the job manager and run the topology from there.

###Ports

• The Web Dashboard is on port 48080
• The Web Client is on port 48081
• JobManager RPC port 6123 (default, not exposed to host)
• TaskManagers RPC port 6121 (default, not exposed to host)
• TaskManagers Data port 6122 (default, not exposed to host)
• JobManager SSH 220
• TaskManagers SSH: randomly assigned port, check wih docker ps

Edit the docker-compose.yml file to edit port settings.

docker-flink's Issues

Step 5 : RUN echo 'root:secret' | chpasswd ---> Running in 3e912902a146 chpasswd: (user root) pam_chauthtok() failed, error: System error chpasswd: (line 1, user root) password not changed 2015/08/04 05:15:50 The command [/bin/sh -c echo 'root:secret' | chpasswd] returned a non-zero code: 1

I'm getting the following error when building the base image. Maybe the root Image ubuntu:trusty changed?

Step 5 : RUN echo 'root:secret' | chpasswd ---> Running in 3e912902a146 chpasswd: (user root) pam_chauthtok() failed, error: System error chpasswd: (line 1, user root) password not changed 2015/08/04 05:15:50 The command [/bin/sh -c echo 'root:secret' | chpasswd] returned a non-zero code: 1