twiechert / docker-arm-flink

[ARM-based] Apache Flink cluster deployment in Docker containers using Docker-Compose

#Apache Flink cluster deployment on Docker using Docker-Compose

##Installation ###Install Docker

Install the following ARM build according to https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-%28ARMv6%29-in-four-steps-%28Wheezy%29

###Install ARM-Compose

Install the ARM build of Docker compose (https://github.com/hypriot/arm-compose). Download dep package from https://packagecloud.io/Hypriot/Schatzkiste/packages/debian/wheezy/docker-compose_1.6.0-25_armhf.deb and install using dpkg

###Get the repo

###Build the images

Images are based on Ubuntu Trusty 14.04 and run Supervor to stay alive when running containers.

The base image installs Oracle Java JDK 1.7 and SSH client & server. You can change the SSH password there or add your own key and adjust SSH config.

• Run ./build.sh

###Deploy

• Deploy cluster and see config/setup log output (best run in a screen session)

docker-compose up

• Deploy as a daemon (and return)

docker-compose up -d

• Scale the cluster up or down to N TaskManagers

docker-compose scale taskmanager=<N>

• Access the JobManager node with SSH (exposed on Port 220)

ssh root@localhost -p 220

or on Mac OS X with boot2docker

ssh root@$(boot2docker ip) -p 220

The password is 'secret'

• Kill the cluster

docker-compose kill

• Upload a jar to the cluster

scp -P 220 <your_jar> root@localhost:/<your_path>

• Run a topology

ssh -p 220 root@localhost /usr/local/flink/bin/flink run -c <your_class> <your_jar> <your_params>

or

ssh to the job manager and run the topology from there.

###Ports

• The Web Dashboard is on port 48080
• The Web Client is on port 48081
• JobManager RPC port 6123 (default, not exposed to host)
• TaskManagers RPC port 6121 (default, not exposed to host)
• TaskManagers Data port 6122 (default, not exposed to host)
• JobManager SSH 220
• TaskManagers SSH: randomly assigned port, check wih docker ps

Edit the docker-compose.yml file to edit port settings.