brooklyncentral / clocker

Apache Brooklyn cloud native infrastructure blueprints

Clocker

The current release is 2.0.0, available in Maven Central. For more information, see the official Clocker site, including documentation and tutorials.

The development version is 2.1.0-SNAPSHOT, available in the Sonatype Open-Source repository. To install this, follow the instructions below.

Overview

Clocker for Apache Brooklyn is a set of open source, Apache Licensed tools designed to make working with Docker containers as simple as a few clicks. Clocker contains Brooklyn blueprints to enable deployment and management of Docker Swarm and Kubernetes clusters.

You will find the source code for the blueprints in this repository.

• Docker
• Swarm
• Kubernetes

Getting Started

Add Clocker to Brooklyn (Karaf Edition)

Add catalog entries using the YAML below:

brooklyn.catalog:
  brooklyn.libraries:
    - "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.brooklyn.etcd&a=brooklyn-etcd&v=2.7.0-SNAPSHOT"
    - "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.brooklyn.clocker&a=clocker-common&v=2.1.0-SNAPSHOT"
    - "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.brooklyn.clocker&a=clocker-swarm&v=2.1.0-SNAPSHOT"
    - "https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=io.brooklyn.clocker&a=clocker-kubernetes&v=2.1.0-SNAPSHOT"
  items:
    - classpath://io.brooklyn.clocker.swarm:swarm/catalog.bom
    - classpath://io.brooklyn.clocker.kubernetes:kubernetes/catalog.bom

Add Clocker to Brooklyn (Classic Edition)

You must add the following JARs to ./lib/dropins:

• brooklyn-etcd
• common
• swarm
• kubernetes

Then add the catalog entries using the following YAML:

brooklyn.catalog:
  items:
    - classpath://swarm/catalog.bom
    - classpath://kubernetes/catalog.bom

Copyright

The following icon images have been used, and their use here is believed to be acceptable under their licensing terms or fair use doctrine. The source URLs and other links given contain more details:

• common/resources/icons/centos.png Remixed version of freely available logo
• common/resources/icons/docker.png Icon taken from media kit
• common/resources/icons/haproxy.png Fair use of logo from Twitter account for open source project
• common/resources/icons/openssl.png Public domain image
• swarm/resources/icons/swarm.png Fair use of image from https://www.docker.com/products/docker-swarm and also found in Docker media kit
• kubernetes/resources/icons/calico.png APACHE 2.0 https://github.com/projectcalico/calico/blob/master/LICENSE image
• kubernetes/resources/icons/flannel.png APACHE 2.0 https://github.com/coreos/flannel/blob/master/LICENSE image
• kubernetes/resources/icons/kubernetes.png APACHE 2.0 https://github.com/kubernetes/kubernetes/blob/master/LICENSE image
• kubernetes/resources/icons/prometheus.png APACHE 2.0 https://github.com/prometheus/docs/blob/master/LICENSE image
• kubernetes/resources/icons/openshift.png APACHE 2.0 https://github.com/openshift/origin-web-console/blob/master/LICENSE image

clocker's Issues

NoSuchMethodError DockerTemplateOptions.hostname(Ljava/lang/String;)

Cloned and built Clocker on a VM in Softlayer Amsterdam and run, again in Amsterdam.

Clocker deploys fine but all attempts to provision containers fail immediately with this exception:

Caused by: java.lang.NoSuchMethodError: org.jclouds.docker.compute.options.DockerTemplateOptions.hostname(Ljava/lang/String;)Lorg/jclouds/docker/compute/options/DockerTemplateOptions;
    at brooklyn.entity.container.docker.DockerContainerImpl.getDockerTemplateOptions(DockerContainerImpl.java:271)
    at brooklyn.entity.container.docker.DockerContainerImpl.createLocation(DockerContainerImpl.java:421)
    at brooklyn.entity.container.docker.DockerContainerImpl.start(DockerContainerImpl.java:570)

Tried experimenting with the Groovy console:

Class.forName("org.jclouds.docker.compute.options.DockerTemplateOptions").getMethod("hostname", String.class)

public org.jclouds.docker.compute.options.DockerTemplateOptions org.jclouds.docker.compute.options.DockerTemplateOptions.hostname(java.lang.String)

DockerTemplateOptions exists in brooklyn-clocker-docker-0.9.0-SNAPSHOT.jar and docker-1.9.0.jar. Both are on the server's classpath.

javap and grepping for hostname on both:

sam@br-vnc5-sam-opil-emp-lv0x-a18:~/git/clocker/examples/target/brooklyn-clocker-dist/brooklyn-clocker/lib/brooklyn$ javap -classpath ./brooklyn-clocker-docker-0.9.0-SNAPSHOT.jar org.jclouds.docker.compute.options.DockerTemplateOptions | grep -i hostname
  protected com.google.common.base.Optional<java.lang.String> hostname;
  public org.jclouds.docker.compute.options.DockerTemplateOptions hostname(java.lang.String);
  public com.google.common.base.Optional<java.lang.String> getHostname();
sam@br-vnc5-sam-opil-emp-lv0x-a18:~/git/clocker/examples/target/brooklyn-clocker-dist/brooklyn-clocker/lib/brooklyn$ javap -classpath docker-1.9.0.jar org.jclouds.docker.compute.options.DockerTemplateOptions | grep -i hostname
  protected com.google.common.base.Optional<java.lang.String> hostname;
  public org.jclouds.compute.options.TemplateOptions hostname(java.lang.String);
  public com.google.common.base.Optional<java.lang.String> getHostname();

I'm struggling to work this one out.

tomcat-cluster-with-mysql.yaml deploy fails: Cannot coerce or set brooklyn.location.docker.strategy.BreadthFirstPlacementStrategy

When deploying clocker/examples/src/main/assembly/files/blueprints/tomcat-cluster-with-mysql.yaml, it failed - the YAML wasn't even accepted to deploy the app. The log file showing the exception below.

2014-11-24 16:06:31,547 WARN  b.m.i.LocalEntityManager [brooklyn-execmanager-OLAp73yN-498]: Failed to create entity using spec EntitySpec{type=interface brooklyn.entity.webapp.DynamicWebAppCluster} (rethrowing)
java.lang.IllegalArgumentException: Cannot coerce or set brooklyn.location.docker.strategy.BreadthFirstPlacementStrategy to dynamiccluster.zone.placementStrategy[ConfigKey:brooklyn.entity.group.DynamicCluster$NodePlacementStrategy]
        at brooklyn.entity.basic.EntityConfigMap.setConfig(EntityConfigMap.java:204) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.basic.AbstractEntity.setConfigInternal(AbstractEntity.java:941) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.basic.AbstractEntity.setConfig(AbstractEntity.java:950) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.proxying.InternalEntityFactory.loadUnitializedEntity(InternalEntityFactory.java:252) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.proxying.InternalEntityFactory.createEntityAndDescendantsUninitialized(InternalEntityFactory.java:202) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.proxying.InternalEntityFactory.createEntity(InternalEntityFactory.java:177) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.management.internal.LocalEntityManager.createEntity(LocalEntityManager.java:146) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.management.internal.NonDeploymentEntityManager.createEntity(NonDeploymentEntityManager.java:58) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.basic.AbstractEntity.addChild(AbstractEntity.java:622) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.webapp.ControlledDynamicWebAppClusterImpl.init(ControlledDynamicWebAppClusterImpl.java:117) [brooklyn-software-webapp-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.proxying.InternalEntityFactory$1.run(InternalEntityFactory.java:294) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_71]
        at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469) [brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.ClassCastException: brooklyn.location.docker.strategy.BreadthFirstPlacementStrategy cannot be cast to brooklyn.entity.group.DynamicCluster$NodePlacementStrategy
        at brooklyn.entity.group.DynamicClusterImpl$1.apply(DynamicClusterImpl.java:97) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.group.DynamicClusterImpl$1.apply(DynamicClusterImpl.java:1) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.util.flags.TypeCoercions.coerce(TypeCoercions.java:241) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        at brooklyn.entity.basic.EntityConfigMap.setConfig(EntityConfigMap.java:202) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
        ... 16 common frames omitted

Execution failed, invalid result 1 for customizing MySqlNodeImpl{id=b25OXMAx}

I am new to clocker and brooklyn I just followed this link:
https://people.apache.org/~richard/quickstart/

But after running the YAML code, i get this error:
Execution failed, invalid result 1 for customizing MySqlNodeImpl{id=b25OXMAx}

in the file "~/.brooklyn/brooklyn.properties" I commented:

brooklyn.location.jclouds.aws-ec2.identity = AKA_YOUR_ACCESS_KEY_ID

brooklyn.location.jclouds.aws-ec2.credential =

and uncommented:
brooklyn.location.localhost.privateKeyFile = ~/.ssh/id_rsa

Clocker install fails on CentOS 6.4 ("Cannot retrieve metalink for repository: epel")

Clocker install failed on CentOS 6.4 in a vcloud-director private environment. The fundamental error appears to be when running yum install docker-io:

Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

Below are snippets from the log:

2014-11-28 21:32:00,450 DEBUG brooklyn.SSH [brooklyn-execmanager-vkWlpua0-1430]: installing DockerHostImpl{id=Czuc5GlD}, initiating ssh on machine SshMachineLocation[vm-612125784:172.16.120.48/172.16.120.48@IcL6tlN0]: #!/bin/bash -e
 ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/DockerHost_1.2" ; mkdir -p $INSTALL_DIR ; cd $INSTALL_DIR ; test -f BROOKLYN && exit 0 ; { { test -z `which yum` && true ; } || { { which wget || ( { which zypper && { echo zypper exists, doing refresh && (( if test "$UID" -eq 0; then ( zypper --non-interactive --no-gpg-checks refresh ); else sudo -E -n -S -- zypper --non-interactive --no-gpg-checks refresh; fi ) || true) && ( if test "$UID" -eq 0; then ( zypper --non-interactive --no-gpg-checks install wget ); else sudo -E -n -S -- zypper --non-interactive --no-gpg-checks install wget; fi ) ; } ; } || { which apt-get && { echo apt-get exists, doing update && export DEBIAN_FRONTEND=noninteractive && (( if test "$UID" -eq 0; then ( apt-get update ); else sudo -E -n -S -- apt-get update; fi ) || true) && ( if test "$UID" -eq 0; then ( apt-get install -y --allow-unauthenticated wget ); else sudo -E -n -S -- apt-get install -y --allow-unauthenticated wget; fi ) ; } ; } || { which yum && { echo yum exists, doing update && (( if test "$UID" -eq 0; then ( yum check-update ); else sudo -E -n -S -- yum check-update; fi ) || true) && ( if test "$UID" -eq 0; then ( yum -y --nogpgcheck install wget ); else sudo -E -n -S -- yum -y --nogpgcheck install wget; fi ) ; } ; } || { which brew && brew install wget ; } || { which port && ( if test "$UID" -eq 0; then ( port install wget ); else sudo -E -n -S -- port install wget; fi ) ; } || (( echo "WARNING: no known/successful package manager to install wget, may fail subsequently" | tee /dev/stderr ) || true) ) ; } && ( ( if test "$UID" -eq 0; then ( rpm -qa | grep epel-release ); else sudo -E -n -S -- rpm -qa | grep epel-release; fi ) || ( if test "$UID" -eq 0; then ( rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm ); else sudo -E -n -S -- rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm; fi ) ) ; } ; } ; ( { which yum && { echo yum exists, doing update && (( if test "$UID" -eq 0; then ( yum check-update ); else sudo -E -n -S -- yum check-update; fi ) || true) && ( if test "$UID" -eq 0; then ( yum -y --nogpgcheck install docker-io ); else sudo -E -n -S -- yum -y --nogpgcheck install docker-io; fi ) ; } ; } || (( echo "WARNING: no known/successful package manager to install {yum=docker-io}, may fail subsequently" | tee /dev/stderr ) || true) ) ; date > $INSTALL_DIR/BROOKLYN
2014-11-28 21:32:01,451 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] /usr/bin/wget
2014-11-28 21:32:05,452 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
2014-11-28 21:32:05,452 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Preparing...                ##################################################
2014-11-28 21:32:05,452 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] epel-release                ##################################################
2014-11-28 21:32:05,455 DEBUG brooklyn.SSH [Thread-44982]: [Czuc5GlD@vm-612125784:stderr] warning: /var/tmp/rpm-tmp.hXLTmx: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
2014-11-28 21:32:06,452 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] /usr/bin/yum
2014-11-28 21:32:06,452 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] yum exists, doing update
2014-11-28 21:32:09,453 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Loaded plugins: fastestmirror, security
2014-11-28 21:32:09,453 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Loading mirror speeds from cached hostfile
2014-11-28 21:32:09,456 DEBUG brooklyn.SSH [Thread-44982]: [Czuc5GlD@vm-612125784:stderr] Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again
2014-11-28 21:32:10,178 DEBUG brooklyn.SSH [Thread-44982]: [Czuc5GlD@vm-612125784:stderr] Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again
2014-11-28 21:32:10,178 DEBUG brooklyn.SSH [Thread-44982]: [Czuc5GlD@vm-612125784:stderr] WARNING: no known/successful package manager to install {yum=docker-io}, may fail subsequently
2014-11-28 21:32:10,179 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Loaded plugins: fastestmirror, security
2014-11-28 21:32:10,179 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Loading mirror speeds from cached hostfile
2014-11-28 21:32:10,179 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] WARNING: no known/successful package manager to install {yum=docker-io}, may fail subsequently
2014-11-28 21:32:10,179 DEBUG brooklyn.SSH [Thread-44981]: [Czuc5GlD@vm-612125784:stdout] Executed /tmp/brooklyn-20141128-213200652-nEQ6-installing_DockerHostImpl_id_C.sh, result 0
2014-11-28 21:32:10,179 DEBUG brooklyn.SSH [brooklyn-execmanager-vkWlpua0-1430]: installing DockerHostImpl{id=Czuc5GlD}, on machine SshMachineLocation[vm-612125784:172.16.120.48/172.16.120.48@IcL6tlN0], completed: return status 0
...

2014-11-28 21:32:10,184 DEBUG brooklyn.SSH [brooklyn-execmanager-vkWlpua0-1342]: stopping DockerHostImpl{id=Czuc5GlD}, initiating ssh on machine SshMachineLocation[vm-612125784:172.16.120.48/172.16.120.48@IcL6tlN0]: #!/bin/bash -e
 ; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/C7EFsQD6/entities/DockerHost_Czuc5GlD" ; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ( { which boot2docker && boot2docker down ; } || { which service && ( if test "$UID" -eq 0; then ( service docker stop ); else sudo -E -n -S -- service docker stop; fi ) ; } )
2014-11-28 21:32:10,339 DEBUG brooklyn.SSH [Thread-45234]: [Czuc5GlD@vm-612125784:stdout] /sbin/service
2014-11-28 21:32:10,339 DEBUG brooklyn.SSH [Thread-45234]: [Czuc5GlD@vm-612125784:stdout] Executed /tmp/brooklyn-20141128-213210219-gkDi-stopping_DockerHostImpl_id_Czu.sh, result 1
2014-11-28 21:32:10,339 DEBUG brooklyn.SSH [Thread-45235]: [Czuc5GlD@vm-612125784:stderr] which: no boot2docker in (/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/root/jdk1.7.0_25/bin:/home/users/amp/bin)
2014-11-28 21:32:10,339 DEBUG brooklyn.SSH [Thread-45235]: [Czuc5GlD@vm-612125784:stderr] docker: unrecognized service
2014-11-28 21:32:10,339 DEBUG brooklyn.SSH [brooklyn-execmanager-vkWlpua0-1342]: stopping DockerHostImpl{id=Czuc5GlD}, on machine SshMachineLocation[vm-612125784:172.16.120.48/172.16.120.48@IcL6tlN0], completed: return status 1
2014-11-28 21:32:10,340 WARN  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-vkWlpua0-1342]: Execution failed, invalid result 1 for stopping DockerHostImpl{id=Czuc5GlD} (throwing)
2014-11-28 21:32:10,340 INFO  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-vkWlpua0-1342]: STDERR of problem in Task[ssh: stopping DockerHostImpl{id=Czuc5GlD}:XjR8cZV8]:
2014-11-28 21:32:10,340 INFO  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-vkWlpua0-1342]: STDOUT of problem in Task[ssh: stopping DockerHostImpl{id=Czuc5GlD}:XjR8cZV8]:
2014-11-28 21:32:10,340 INFO  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-vkWlpua0-1342]: STDIN of problem in Task[ssh: stopping DockerHostImpl{id=Czuc5GlD}:XjR8cZV8]:
export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/C7EFsQD6/entities/DockerHost_Czuc5GlD"

I then tried running the commands manually on the VM that install had failed on:

[amp@Centos64x64 ~]$ sudo rpm -qa | grep epel-release
epel-release-6-8.noarch

[amp@Centos64x64 ~]$ curl -O http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14540  100 14540    0     0  27847      0 --:--:-- --:--:-- --:--:-- 48791

[amp@Centos64x64 ~]$ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
warning: /var/tmp/rpm-tmp.IAux1P: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
    package epel-release-6-8.noarch is already installed

[amp@Centos64x64 ~]$ sudo yum -y --nogpgcheck install docker-io
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
Error: Cannot retrieve metalink for repository: epel. Please verify its path and try again

Fails at startup then reboots machine

Trying to use 0.8.1 on ubuntu but it restarts the machine. Here's a Vagrant file to recreate:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision :shell, path: "bootstrap.sh"
  config.vm.network :forwarded_port, host: 8888, guest: 8081
  config.vm.synced_folder ".", "/root/.brooklyn", mount_options: ["dmode=777", "fmode=600"]
  config.vm.provider "virtualbox" do |v|
    v.memory = 1048
  end
end

with bootstrap.sh

sudo apt-get update -qq
sudo apt-get install -qqy openjdk-7-jdk

ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

wget --no-check-certificate --quiet \
  -O brooklyn-clocker-dist.tar.gz http://git.io/vfCE8
tar zxf brooklyn-clocker-dist.tar.gz
cd brooklyn-clocker
./bin/clocker.sh localhost weave

I get this output, before the VM reboots

 _                     _    _             
| |__  _ __ ___   ___ | | _| |_   _ _ __ (R)
| '_ \| '__/ _ \ / _ \| |/ / | | | | '_ \ 
| |_) | | | (_) | (_) |   <| | |_| | | | |
|_.__/|_|  \___/ \___/|_|\_\_|\__, |_| |_|
                              |___/             0.7.0-ea-cloudsoft.7

2015-05-19 13:05:40,785 INFO  YAML app combined with command line locations; YAML locations will take precedence; this behaviour may change in subsequent versions
2015-05-19 13:05:42,903 INFO  Persistence disabled
2015-05-19 13:05:42,904 INFO  High availability disabled
2015-05-19 13:05:46,615 INFO  Geo info lookup unavailable (for 127.0.0.1/127.0.0.1; cause brooklyn.util.exceptions.RuntimeInterruptedException: java.lang.InterruptedException)
2015-05-19 13:05:46,632 INFO  Starting Brooklyn web-console with passwordless access on localhost and protected access from any other interfaces (no bind address specified)
2015-05-19 13:05:48,645 INFO  Started Brooklyn console at http://127.0.0.1:8081/, running brooklyn-jsgui.war and [brooklyn-jsgui.war, brooklyn-clocker-console.war]
2015-05-19 13:05:49,054 INFO  Starting Docker infrastructure id DwF0lsgU
2015-05-19 13:05:49,240 INFO  Starting Weave network id bWRNjUDl
2015-05-19 13:05:49,240 INFO  Starting SDN provider id bWRNjUDl
2015-05-19 13:05:49,529 INFO  Starting brooklyn application BasicApplicationImpl{id=wHegkoCP} in location [LocalhostMachineProvisioningLocation{id=jKm2eygh, name=localhost}]
2015-05-19 13:05:49,628 INFO  Creating new DockerLocation wrapping LocalhostMachineProvisioningLocation{id=jKm2eygh, name=localhost}
2015-05-19 13:05:49,642 INFO  New Docker location DockerLocation{id=gYaMrzJA, name=Docker Infrastructure DwF0lsgU, provisioner=LocalhostMachineProvisioningLocation{id=jKm2eygh, name=localhost}, infrastructure=DockerInfrastructureImpl{id=DwF0lsgU}, strategies=[DockerAwarePlacementStrategy(MaxContainersPlacementStrategy@vtN2OPDa), DockerAwarePlacementStrategy(BreadthFirstPlacementStrategy@KIYkjc8I)]} created
2015-05-19 13:05:49,645 INFO  Starting entity DockerInfrastructureImpl{id=DwF0lsgU} at [LocalhostMachineProvisioningLocation{id=jKm2eygh, name=localhost}]
2015-05-19 13:05:49,719 INFO  Starting entity WeaveNetworkImpl{id=bWRNjUDl} at [LocalhostMachineProvisioningLocation{id=jKm2eygh, name=localhost}]
2015-05-19 13:05:49,747 INFO  Added policy MemberTrackingPolicy{name=Docker host tracker, running=true} to WeaveNetworkImpl{id=bWRNjUDl}, during start
2015-05-19 13:05:49,752 INFO  Resize DynamicClusterImpl{id=pUBUQvAC} from 0 to 2
2015-05-19 13:05:49,839 INFO  Starting Docker host id IuE3vjGd
2015-05-19 13:05:49,839 INFO  Starting server pool machine with id IuE3vjGd
2015-05-19 13:05:49,965 INFO  Starting Docker host id GjwKQ9XD
2015-05-19 13:05:49,965 INFO  Starting server pool machine with id GjwKQ9XD
2015-05-19 13:05:50,113 INFO  Starting DockerHostImpl{id=GjwKQ9XD} on machine SshMachineLocation[LocalhostMachine:RJXF:127.0.0.1/127.0.0.1:22@RJXFN5Ia]
2015-05-19 13:05:50,125 INFO  Starting DockerHostImpl{id=IuE3vjGd} on machine SshMachineLocation[LocalhostMachine:ZeXT:127.0.0.1/127.0.0.1:22@ZeXT7XWU]
2015-05-19 13:05:50,273 WARN  DockerHostImpl{id=GjwKQ9XD}: port 2375 not available for docker.port
2015-05-19 13:05:50,278 WARN  DockerHostImpl{id=GjwKQ9XD}: port 2376 not available for docker.ssl.port
2015-05-19 13:05:50,280 INFO  DockerHostImpl{id=IuE3vjGd} not running in a JcloudsSshMachineLocation, not configuring extra security groups
2015-05-19 13:05:50,281 INFO  DockerHostImpl{id=GjwKQ9XD} not running in a JcloudsSshMachineLocation, not configuring extra security groups
2015-05-19 13:05:50,406 WARN  Deprecated use of unmanaged location (DockerHostLocation{id=eWbyYHmZ, name=Docker Host GjwKQ9XD, machine=SshMachineLocation[LocalhostMachine:RJXF:127.0.0.1/127.0.0.1:22@RJXFN5Ia], jcloudsLocation=JcloudsLocation[docker:https://127.0.0.1:null:conf/server-cert.pem/docker:https://127.0.0.1:null@bEQ0TlUm], dockerHost=DockerHostImpl{id=GjwKQ9XD}}); will be managed automatically now but not supported in future versions
2015-05-19 13:05:50,414 WARN  Deprecated use of unmanaged location (DockerHostLocation{id=M65pH8nK, name=Docker Host IuE3vjGd, machine=SshMachineLocation[LocalhostMachine:ZeXT:127.0.0.1/127.0.0.1:22@ZeXT7XWU], jcloudsLocation=JcloudsLocation[docker:https://127.0.0.1:2376:conf/server-cert.pem/docker:https://127.0.0.1:2376@wCaLcufg], dockerHost=DockerHostImpl{id=IuE3vjGd}}); will be managed automatically now but not supported in future versions
2015-05-19 13:05:50,420 INFO  New Docker host location DockerHostLocation{id=eWbyYHmZ, name=Docker Host GjwKQ9XD, machine=SshMachineLocation[LocalhostMachine:RJXF:127.0.0.1/127.0.0.1:22@RJXFN5Ia], jcloudsLocation=JcloudsLocation[docker:https://127.0.0.1:null:conf/server-cert.pem/docker:https://127.0.0.1:null@bEQ0TlUm], dockerHost=DockerHostImpl{id=GjwKQ9XD}} created
2015-05-19 13:05:50,421 INFO  New Docker host location DockerHostLocation{id=M65pH8nK, name=Docker Host IuE3vjGd, machine=SshMachineLocation[LocalhostMachine:ZeXT:127.0.0.1/127.0.0.1:22@ZeXT7XWU], jcloudsLocation=JcloudsLocation[docker:https://127.0.0.1:2376:conf/server-cert.pem/docker:https://127.0.0.1:2376@wCaLcufg], dockerHost=DockerHostImpl{id=IuE3vjGd}} created
2015-05-19 13:05:52,075 WARN  SSH task ended with exit code 1 when non-zero was not explicitly allowed (error may be thrown in future), in Task[ssh: modprobe overlay:lC0JchRt]: modprobe overlay
2015-05-19 13:05:52,103 WARN  SSH task ended with exit code 1 when non-zero was not explicitly allowed (error may be thrown in future), in Task[ssh: modprobe overlay:mRLlW0X9]: modprobe overlay

Broadcast message from root@vagrant-ubuntu-trusty-64
    (unknown) at 13:06 ...

The system is going down for reboot NOW!
Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.```

Entities fail if zero hosts started

Failure layering SshableImage

Running from master with #123. I ran ./clocker.sh jclouds:softlayer:ams01. Clocker deployed successfully. I tried to deploy docker-jboss-wildfly.yaml but the layerSshableImageOn task failed. This was the stdout:

Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon
Step 0 : FROM jboss/wildfly:8.2.0.Final
Pulling repository jboss/wildfly
c1abaf3a478a: Pulling image (8.2.0.Final) from jboss/wildfly
c1abaf3a478a: Pulling image (8.2.0.Final) from jboss/wildfly, endpoint: https://registry-1.docker.io/v1/
c1abaf3a478a: Pulling dependent layers
...
c1abaf3a478a: Download complete
Status: Downloaded newer image for jboss/wildfly:8.2.0.Final
 ---> c1abaf3a478a
Step 1 : MAINTAINER Cloudsoft "[email protected]"
 ---> Running in 3b689ecf8e47
 ---> faeb35a56528
Removing intermediate container 3b689ecf8e47
Step 2 : RUN type locale-gen ; if [ "$?" -eq "0" ] ; then locale-gen en_US.UTF-8 ; fi
 ---> Running in cf27628fd9bf
�[91m/bin/sh: line 0: type: locale-gen: not found
�[0m ---> 143fa3bfd1de
Removing intermediate container cf27628fd9bf
Step 3 : RUN echo 'root:hHWPXilJ' | chpasswd
 ---> Running in a10116147506
�[91mchpasswd: Permission denied.
�[0m�[91mchpasswd: cannot lock /etc/passwd; try again later.
�[0m

Clocker on GCE failed to deploy

I tried to start clocker on GCE, but it failed (and it created several additional entities+VMs when retrying).

I downloaded the latest tgz from clocker.io's big red button, and then ran ./bin/clocker.sh google-compute-engine:us-central1-a.

I'll share the huge log with @grkvlt who can update this issue with additional details.

Scale out of Docker Hosts - Can't see activities

https://www.dropbox.com/s/pmleyfta2oyenb6/Screenshot%202014-11-04%2016.11.05.png?dl=0

Occasional error stopping containers

Every so often Brooklyn throws this exception when stopping containers:

2014-04-23 16:21:40,185 ERROR Cannot retry after server error, command has exceeded retry limit 15: [method=public abstract void org.jclouds.docker.compute.features.RemoteApi.removeContainer(java.lang.String)[320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29], request=DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1]
2014-04-23 16:21:40,188 ERROR Problem releasing machine SshMachineLocation[null:127.0.0.1/127.0.0.1] in JcloudsPortforwardingSubnetLocation[docker:http://127.0.0.1:4243:notused], instance id 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29; discarding instance and continuing...
org.jclouds.http.HttpResponseException: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
    at org.jclouds.docker.handlers.DockerErrorHandler.handleError(DockerErrorHandler.java:49) ~[docker-1.7.2-ea-brooklyn.5.jar:1.7.2-ea-brooklyn.5]
2014-04-23 16:21:40,195 WARN  Error invoking stop at DataFullStackImpl{id=AfIKmcWN}: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
2014-04-23 16:21:40,199 INFO  TESTNG FAILED: "Surefire test" - io.cloudsoft.margining.entity.DataFullStackIntegrationTest.testCanStartAndStop() finished in 86938 ms
brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at DataFullStackImpl{id=AfIKmcWN}: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
    at brooklyn.management.internal.EffectorUtils.handleEffectorException(EffectorUtils.java:231)
    at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:223)
    at brooklyn.entity.proxying.EntityProxyImpl.invoke(EntityProxyImpl.java:84)
    at com.sun.proxy.$Proxy22.stop(Unknown Source)
    at io.cloudsoft.margining.entity.DataFullStackIntegrationTest.testCanStartAndStop(DataFullStackIntegrationTest.java:39)
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: brooklyn.util.exceptions.PropagatedRuntimeException: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:231)
    at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:218)
    ... 31 more
Caused by: java.util.concurrent.ExecutionException: brooklyn.util.exceptions.PropagatedRuntimeException: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
    at brooklyn.util.task.BasicTask.get(BasicTask.java:304)
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:225)
    ... 32 more
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException: command: DELETE http://127.0.0.1:4243/containers/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 HTTP/1.1 failed with response: HTTP/1.1 500 Internal Server Error; content: [Cannot destroy container 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: Driver aufs failed to remove root filesystem 320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29: rename /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29 /mnt/sda1/var/lib/docker/aufs/mnt/320e67287a050ac694e807a9e0936761b38a02c5c0e3b21237eb1b4c66c04e29-removing: device or resource busy
]
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:70)
    at brooklyn.util.task.BasicTask.getUnchecked(BasicTask.java:315)
    at brooklyn.entity.basic.SoftwareProcessImpl.stop(SoftwareProcessImpl.java:400)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1047)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
    at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
    at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
    at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:125)
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:204)
    at brooklyn.management.internal.AbstractManagementContext$2.call(AbstractManagementContext.java:223)
    at brooklyn.util.task.BasicExecutionManager$3.call(BasicExecutionManager.java:352)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
...

Containers do appear to be stopped, but are not removed as I believe cleanly shutdown container are (i.e. they are listed by docker ps -a).

Location definitions created by the DockerInfrastructureImpl are not removed

When the DockerInfrastructureImpl.createLocation method is called a new BasicLocationDefinition is created and registered with the location registry. This is not removed when DockerInfrastructureImpl.deleteLocation is called. I think the most sensible way to fix this changes Clocker and Brooklyn Core. Adding a LOCATION_DEFINITION attribute to LocationOwner setting it when creating the location and using it to unregister the location definition when removing the location.

Installation erroneously marked complete

Clocker marks Docker host installation as complete (by touching installDir/BROOKLYN) even when Docker couldn't be installed because the user cannot sudo. This means that Clocker will never reattempt installation until the directory is deleted and means that Clocker will not work even if the sudo permissions error is fixed.

trouble building from src

Hi, I'm trying to build from src by following the instruction from the readme but yield the error below, looks like missing lib it depends on. I'm not very familiar with maven so it is possible that my machine lacks some setup.

here's what I got by doing the following:

my machine config: (Ubuntu 14.04.1)

Apache Maven 3.0.5
Maven home: /usr/share/maven
Java version: 1.8.0_25, vendor: Oracle Corporation
Java home: /usr/lib/jvm/jdk1.8.0_25/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-37-generic", arch: "amd64", family: "unix"

% git clone https://github.com/brooklyncentral/clocker.git
...
% cd clocker
% mvn clean install

...
...

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project brooklyn-clocker-docker: Compilation failure: Compilation failure:
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/location/docker/strategy/affinity/DockerAffinityRuleStrategy.java:[53,41] cannot find symbol
[ERROR] symbol: method getBrooklynManagementContext()
[ERROR] location: class brooklyn.location.docker.strategy.affinity.DockerAffinityRuleStrategy
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/location/docker/strategy/ProvisioningFlagsPlacementStrategy.java:[44,72] incompatible types: brooklyn.config.ConfigKey<java.lang.Integer> cannot be converted to brooklyn.config.ConfigKey<java.lang.Object>
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/location/docker/strategy/ProvisioningFlagsPlacementStrategy.java:[70,45] cannot find symbol
[ERROR] symbol: method getBrooklynManagementContext()
[ERROR] location: class brooklyn.location.docker.strategy.ProvisioningFlagsPlacementStrategy
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/entity/container/weave/WeaveContainerSshDriver.java:[86,69] cannot find symbol
[ERROR] symbol: variable FIRST_MEMBER
[ERROR] location: interface brooklyn.entity.basic.AbstractGroup
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/entity/container/weave/WeaveContainerSshDriver.java:[87,62] cannot find symbol
[ERROR] symbol: variable FIRST
[ERROR] location: interface brooklyn.entity.basic.AbstractGroup
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/entity/container/docker/DockerHostImpl.java:[479,5] method does not override or implement a method from a supertype
[ERROR] /home/shiqi/workspace/clocker-dev/clocker/docker/src/main/java/brooklyn/entity/container/docker/DockerHostImpl.java:[481,14] cannot find symbol
[ERROR] symbol: method postStop()
[ERROR] -> [Help 1]

Testing on GC Platform

Hi @andreaturli
I have ran some test on different machines on GCloudPlatform, on Debian and Centos7 with two example nodes, but I've got errors. I include here the logfile

https://gist.github.com/anonymous/b759df53d6891e6cad4e#file-gistfile1-txt

nodes are correctly sshable and
brooklyn.properties contains few rows

brooklyn.webconsole.security.users=admin
brooklyn.webconsole.security.user.admin.password=password

brooklyn.location.named.gcpool=byon:(hosts="199.223.233.216")

Incorrect logo in "Clocker with Weave"

If you go to the Brooklyn web console, you can create "Local Clocker" which has a Docker logo; "Clocker with Calico" which has a Calico logo; and "Clocker with Weave" which has the Docker logo again. That last probably should have the Weave logo (although I confess that I have been unable to find a weave logo that is definitively the right one to use!).

Affinity

Add affinity and anti-affinity API and DSL when selecting DockerHostLocations to add new containers

calico.sh usage warning is wrong

This is rather a trivial issue, but the calico.sh script contains a usage statement that says "Usage: clocker.sh...".

I'd just submit a patch, but given that clocker.sh and calico.sh are exactly the same except for one file name, it occurred to me to ask if instead it would be better to have just a single clocker.sh, and present the networking type as an argument to it? That seems more intuitive to me, since calico.sh is an odd command to run to start up clocker!

Headroom enricher goes haywire when no more locations can be provisioned

Relevant config:

    docker.host.cluster.initial.size: 2
    docker.container.cluster.headroom.count: 6

I ran Clocker in a FixedListMachineProvisioningLocation that had a single location. This is obviously incorrect and one of the DockerHosts was not provisioned. Thirty seconds after the application launch had finished the headroom enricher decided to take matters into its own hands:

<startup>
2015-01-30 11:34:27,074 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-9]: Headroom enricher: 0 containers on 1 hosts, 6 available and 0 needed
2015-01-30 11:34:27,190 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-12]: Headroom enricher: 0 containers on 2 hosts, 12 available and -6 needed
2015-01-30 11:35:29,562 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-34]: Headroom enricher: 0 containers on 1 hosts, 6 available and 0 needed
2015-01-30 11:35:29,921 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-7]: Headroom enricher: 1 containers on 1 hosts, 5 available and 1 needed
<haywire>
2015-01-30 11:36:00,081 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-51]: Headroom enricher: 1 containers on 2 hosts, 11 available and -5 needed
2015-01-30 11:36:00,153 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-49]: Headroom enricher: 1 containers on 3 hosts, 17 available and -11 needed
... ten seconds and several hundred lines later ...
2015-01-30 11:36:11,941 DEBUG b.e.c.p.ContainerHeadroomEnricher [brooklyn-execmanager-lkn8knca-49]: Headroom enricher: 1 containers on 226 hosts, 1355 available and -1349 needed

Then I killed Brooklyn.

I've removed them for brevity but each line was logged at least twice, some three times.

The enricher should gracefully handle an inability to gain capacity.

Virtual network groups are not updated when applications are stopped

The SDN Managed Networks and SDN Networked Applications groups are not updated when applications are stopped, leaving useless "Virtual Network" entities behind.

The children of the managed networks group are instances of brooklyn.networking.VirtualNetwork, the children of the applications group are of brooklyn.entity.basic.DynamicGroup.

Rename brooklyn-docker to brooklyn-clocker in logback.xml

We need to rename value=brooklyn-docker into value=brooklyn-clocker in https://github.com/brooklyncentral/clocker/blob/master/examples/src/main/assembly/conf/logback.xml

Container creation fails if Clocker server and Docker hosts running in same AWS region

We deployed a clocker instance to a VM in AWS us-west-1, and then from that we deployed a "Clocker with Weave" to the same AWS region. This deployed successfully.

However, whenever attempting to deploy an app to the new docker-location, it failed to create any containers.

The reason is that the auto-generated security group opens port 2376 just to the public IP of the Clocker server. However, when connecting from within the same region it uses an internal IP address. Therefore the Clocker server cannot reach port 2376 on the Docker hosts (as confirmed using telnet <hostname> 2376 from the Clocker server).

The workaround is to deploy your "Clocker with Weave" to a different AWS region.

Travis CI fails with missing DOCKER_CONTAINER_CLUSTER_MAX_SIZE.

I visited https://travis-ci.org/brooklyncentral/clocker and saw the "Docker Clocker Integration" module failed to build.

Cannot establish ssh connection to root @ SshMachineLocation[LocalhostMachine:oLk6:eb25-2/127.0.1.1] (publickey auth failed).

I can't run docker from clocker on my own machine (localhost).
I am using: brooklyn-clocker-examples-0.4.0
When I run "docker cloud" then "localhost", I get error.

This is what I added to brooklyn.properties:
brooklyn.location.localhost.privateKeyFile = ~/.ssh/id_rsa
brooklyn.ssh.config.privateKeyFile = ~/.ssh/id_rsa
brooklyn.ssh.config.publicKeyFile = ~/.ssh/id_rsa.pub

And this is what I commented:

brooklyn.location.jclouds.aws-ec2.identity = AKA_YOUR_ACCESS_KEY_ID

brooklyn.location.jclouds.aws-ec2.credential =

So I don't understand if clocker is trying to "ssh root@localhost", because that not possible.
From another machine I can "ssh user@IP_address" without password, so it seems ssh key is fine.

Here is full error:
Failure running task invoking start[locations] on 1 node (TTlb0wbr): All nodes in cluster DynamicClusterImpl{id=gRiCX8rp} failed; first failure is: 2 of 2 parallel child tasks failed, including: Cannot establish ssh connection to root @ SshMachineLocation[LocalhostMachine:oLk6:eb25-2/127.0.1.1](publickey auth failed). Ensure that passwordless and passphraseless ssh access is enabled using standard keys from ~/.ssh or as configured in brooklyn.properties. Check that the target host is accessible, that credentials are correct (location and permissions if using a key), that the SFTP subsystem is available on the remote side, and that there is sufficient random noise in /dev/random on both ends. To debug less common causes, see the original error in the trace or log, and/or enable 'net.schmizz' (sshj) logging.

Problem creating Docker Cloud

I've installed 0.6.2 version but some services don't start correctly. The error in dashboard is:

Status ON_FIRE
Service Up false
Type brooklyn.clocker.example.DockerCloud
ID vhpko6ay
DockerInfrastructureImpl{id=yFBYlBwh} is not up
Failure running task invoking start[locations] on 1 node (ObiJFcwt): Node in cluster DynamicClusterImpl{id=pSTRQoyb} failed: 2 of 2 parallel child tasks failed, including: Execution failed, invalid result 1 for installing DockerHostImpl{id=hJGabWgW}

I've in execution Docker 1.3 on Fedora 20

Placement strategies are inconsistent

The behaviour of placement strategies can vary depending on when the entity requesting a container is being created - during scale-out of a running application vs. during deployment when no other entities are created, let alone started inside other containers. Much of the difference is due to the way Brooklyn handles the entity lifecycle, and what knowledge we can obtain about the topology of the entire blueprint at early stages of execution. Solving this is not solely a Clocker issue.

Attributes.MAIN_URI not found.

I cloned the project tonight. When I tried to compile it, I ran into a problem with Attributes.MAIN_URI not being defined.

docker container entity temporarily shown as "on-fire" during provisioning

Docker container entities are sometimes temporarily shown as "on-fire" during provisioning.

My suspicion is that DockerContainerImpl.start() is setting the expected state to Lifecycle.RUNNING before we have actually started the container. We don't set SERVICE_UP until the container is ssh'able, so it is reported as inferred actual state "on-fire" until that point.

I think we should also be setting the service-up-indicators rather than service-up directly. See https://github.com/apache/incubator-brooklyn/blob/master/software/base/src/main/java/brooklyn/entity/basic/SoftwareProcessImpl.java#L131-135 for some pointers into that code.

Below are a few selected log statements for where the docker container entity goes on-fire:

2014-11-24 16:13:38,425 DEBUG b.m.internal.EffectorUtils [brooklyn-execmanager-OLAp73yN-923]: Invoking effector start on DockerContainerImpl{id=ylP0mWEZ}
2014-11-24 16:13:43,846 DEBUG b.event.feed.AbstractFeed [brooklyn-execmanager-OLAp73yN-923]: Starting feed FunctionFeed{running=true, entity=DockerContainerImpl{id=ylP0mWEZ}, id=wB8ddkrX} for DockerContainerImpl{id=ylP0mWEZ}
2014-11-24 16:13:43,852 DEBUG brooklyn.event.feed.Poller [brooklyn-execmanager-OLAp73yN-923]: Starting poll for DockerContainerImpl{id=ylP0mWEZ} (using Poller{entity=DockerContainerImpl{id=ylP0mWEZ}})
2014-11-24 16:13:43,855 INFO  b.e.basic.BasicStartableImpl [brooklyn-execmanager-OLAp73yN-923]: Starting entity DockerContainerImpl{id=ylP0mWEZ} at [SshMachineLocation[ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:ec2-54-220-203-29.eu-west-1.compute.amazonaws.com/54.220.203.29@JHNUXQIi]]
2014-11-24 16:13:44,060 WARN  b.e.basic.ServiceStateLogic [brooklyn-execmanager-OLAp73yN-923]: Service is not up when setting running on DockerContainerImpl{id=ylP0mWEZ}; delayed 204ms but Sensor: service.isUp (java.lang.Boolean) did not recover from false; not-up-indicators=null
2014-11-24 16:13:44,060 WARN  b.e.basic.ServiceStateLogic [brooklyn-execmanager-OLAp73yN-923]: Setting DockerContainerImpl{id=ylP0mWEZ} on-fire due to problems when expected running, up=false, not-up-indicators: null
}, jcloudsParent=JcloudsLocation[docker:http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375:docker/docker:http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375@wMg36Ocq], node={id=f4085fffb9ffda0190f3bf821c3feb39dae52cb105b299e337c311f12fc0b1d8, providerId=f4085fffb9ffda0190f3bf821c3feb39dae52cb105b299e337c311f12fc0b1d8, name=brooklyn-eke3-amp-485, location={scope=PROVIDER, id=docker, description=http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375}, group=brooklyn-eke3-amp, imageId=ff88fdd1edab8de767528583843e6378d36fb9e537b799ac80af2bcef40ccb78, os={family=unrecognized, version=latest, description=rbuhnqst/ejrwgjzbtfg:latest, is64Bit=true}, status=RUNNING, loginPort=49163, hostname=ec2-54-220-203-29.eu-west-1.compute.amazonaws.com, privateAddresses=[172.17.0.12], publicAddresses=[ec2-54-220-203-29.eu-west-1.compute.amazonaws.com], hardware={id=, processors=[{cores=0.0, speed=0.0}], ram=0, supportsImage=Predicates.alwaysTrue()}, loginUser=root}, region=docker, callerContext=null, provider=docker, identity=docker, credential=xxxxxxxx, jclouds.image.login-user=root:rnmjqriK, endpoint=http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375, spec.final=jclouds:docker:http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375, spec.original=jclouds:docker:http://ec2-54-220-203-29.eu-west-1.compute.amazonaws.com:2375, machineCreationSemaphore=java.util.concurrent.Semaphore@be2cac9[Permits = 2147483647], owner=DockerContainerImpl{id=ylP0mWEZ}}
2014-11-24 16:14:29,618 INFO  b.l.d.DockerContainerLocation [brooklyn-execmanager-OLAp73yN-992]: Executing callback for DockerContainerImpl{id=ylP0mWEZ}: subnet-address

Exception parsing JSON when provisioning container

Approximately 10% of containers I attempt to provision fail immediately with this exception:

2015-06-24 10:06:19,793 ERROR o.j.http.functions.ParseJson [brooklyn-execmanager-H8ch5nak-298]: Error parsing input: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 958 path $.HostConfig.Dns
com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 958 path $.HostConfig.Dns
    at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:190) ~[jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:272) ~[jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:184) ~[jclouds-core-1.9.0.jar:1.9.0]
    at com.google.gson.Gson.fromJson(Gson.java:805) ~[gson-2.3.jar:na]
    at com.google.gson.Gson.fromJson(Gson.java:770) ~[gson-2.3.jar:na]
    at com.google.gson.Gson.fromJson(Gson.java:719) ~[gson-2.3.jar:na]
    at org.jclouds.json.internal.GsonWrapper.fromJson(GsonWrapper.java:42) ~[jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:83) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:77) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:62) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:42) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) [jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) [jclouds-core-1.9.0.jar:1.9.0]
    at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) [guava-17.0.jar:na]
    at com.sun.proxy.$Proxy172.inspectContainer(Unknown Source) [na:na]
    at org.jclouds.docker.compute.strategy.DockerComputeServiceAdapter.listNodes(DockerComputeServiceAdapter.java:212) [brooklyn-clocker-docker-0.9.0-SNAPSHOT.jar:na]
    at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listDetailsOnNodesMatching(AdaptingComputeServiceStrategies.java:122) [jclouds-compute-1.9.0.jar:1.9.0]
    at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.listNodes(AdaptingComputeServiceStrategies.java:112) [jclouds-compute-1.9.0.jar:1.9.0]
    at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getNextNames(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:192) [jclouds-compute-1.9.0.jar:1.9.0]
    at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:121) [jclouds-compute-1.9.0.jar:1.9.0]
    at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210) [jclouds-compute-1.9.0.jar:1.9.0]
    at brooklyn.location.jclouds.JcloudsLocation.obtainOnce(JcloudsLocation.java:686) [brooklyn-locations-jclouds-0.7.0-20150615.084013-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.location.jclouds.JcloudsLocation.obtain(JcloudsLocation.java:581) [brooklyn-locations-jclouds-0.7.0-20150615.084013-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.container.docker.DockerContainerImpl.createLocation(DockerContainerImpl.java:453) [brooklyn-clocker-docker-0.9.0-SNAPSHOT.jar:na]
    at brooklyn.entity.container.docker.DockerContainerImpl.start(DockerContainerImpl.java:570) [brooklyn-clocker-docker-0.9.0-SNAPSHOT.jar:na]
    at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_51]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_51]
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) [groovy-all-2.3.4.jar:2.3.4]
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) [groovy-all-2.3.4.jar:2.3.4]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1085) [groovy-all-2.3.4.jar:2.3.4]
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909) [groovy-all-2.3.4.jar:2.3.4]
    at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149) [groovy-all-2.3.4.jar:2.3.4]
    at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source) [groovy-all-2.3.4.jar:2.3.4]
    at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144) [brooklyn-utils-groovy-0.7.0-20150615.084055-253.jar:0.7.0-SNAPSHOT]
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:286) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:310) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:250) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.trait.Startable$StartEffectorBody.call(Startable.java:57) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.trait.Startable$StartEffectorBody.call(Startable.java:51) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:83) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:337) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469) [brooklyn-core-0.7.0-20150615.083911-254.jar:0.7.0-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.lang.IllegalStateException: Expected a string but was BEGIN_ARRAY at line 1 column 958 path $.HostConfig.Dns
    at com.google.gson.stream.JsonReader.nextString(JsonReader.java:836) ~[gson-2.3.jar:na]
    at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:358) ~[gson-2.3.jar:na]
    at com.google.gson.internal.bind.TypeAdapters$13.read(TypeAdapters.java:346) ~[gson-2.3.jar:na]
    at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$ParameterReader.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:272) ~[jclouds-core-1.9.0.jar:1.9.0]
    at org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory$DeserializeIntoParameterizedConstructor.read(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:184) ~[jclouds-core-1.9.0.jar:1.9.0]
    ... 49 common frames omitted

The following is the line of the jclouds log immediately before the exception above was thrown:

2015-06-24 10:06:19,790 DEBUG jclouds.wire [brooklyn-execmanager-H8ch5nak-298]: << "{"AppArmorProfile":"","Args":["-D"],"Config":{"AttachStderr":false,"AttachStdin":false,"AttachStdout":false,"Cmd":["/usr/sbin/sshd","-D"],"CpuShares":0,"Cpuset":"","Domainname":"","Entrypoint":[],"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"ExposedPorts":{"22/tcp":{},"4369/tcp":{},"6000/tcp":{},"7999/tcp":{},"8087/tcp":{},"8093/tcp":{},"8098/tcp":{},"8099/tcp":{},"8985/tcp":{}},"Hostname":"acb3b897cc81","Image":"eeb870a58a73279a845f97b4d351c7fdcea9e9f268dc39b04861f526328e9f95","Labels":{},"MacAddress":"","Memory":0,"MemorySwap":0,"NetworkDisabled":false,"OnBuild":null,"OpenStdin":false,"PortSpecs":null,"StdinOnce":false,"Tty":false,"User":"","Volumes":null,"WorkingDir":""},"Created":"2015-06-24T09:06:19.59166437Z","Driver":"overlay","ExecDriver":"native-0.2","ExecIDs":null,"HostConfig":{"Binds":[],"CapAdd":[],"CapDrop":[],"CgroupParent":"","ContainerIDFile":"","CpuShares":0,"CpusetCpus":"","Devices":null,"Dns":[],"DnsSearch":null,"ExtraHosts":null,"IpcMode":"","Links":null,"LogConfig":{"Config":null,"Type":"json-file"},"LxcConf":[],"Memory":0,"MemorySwap":0,"NetworkMode":"","PidMode":"","PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":""},"SecurityOpt":null,"Ulimits":null,"VolumesFrom":null},"HostnamePath":"","HostsPath":"","Id":"acb3b897cc814fbe38b819a85edfbda14c4aab8e7df25200a4d85db79fc2fef7","Image":"eeb870a58a73279a845f97b4d351c7fdcea9e9f268dc39b04861f526328e9f95","LogPath":"","MountLabel":"","Name":"/brooklyn-pshh-sam-8ac","NetworkSettings":{"Bridge":"","Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","LinkLocalIPv6Address":"","LinkLocalIPv6PrefixLen":0,"MacAddress":"","PortMapping":null,"Ports":null},"Path":"/usr/sbin/sshd","ProcessLabel":"","ResolvConfPath":"","RestartCount":0,"State":{"Dead":false,"Error":"","ExitCode":0,"FinishedAt":"0001-01-01T00:00:00Z","OOMKilled":false,"Paused":false,"Pid":0,"Restarting":false,"Running":false,"StartedAt":"0001-01-01T00:00:00Z"},"Volumes":{},"VolumesRW":{}}[\n]"

Nodes in Calico cluster unable to communicate

I ran Clocker version 1.1.0.PREVIEW_20150904 with Calico.

The cluster started ok (i.e. everything went green). I deployed the Riak web cluster example. The Riak nodes are green but the "joinCluster" effector fails and the cluster entity is on fire.

The effector was executed in a container on a host in AWS with IP 54.177.12.78 and internal address 10.233.137.103. The container's addresses were ["50.0.1.4","50.0.2.4"].

Stdin:

( if test "$UID" -eq 0; then ( riak-admin cluster join [email protected] ); else sudo -E -n -S -- riak-admin cluster join [email protected]; fi )
( if test "$UID" -eq 0; then ( riak-admin cluster plan ); else sudo -E -n -S -- riak-admin cluster plan; fi )
( if test "$UID" -eq 0; then ( riak-admin cluster commit ); else sudo -E -n -S -- riak-admin cluster commit; fi )

Stdout:

Node [email protected] is not reachable!
Executed /tmp/brooklyn-20150908-134159422-NSJ2-joinCluster_RiakNodeImpl_id_r2.sh, result 1

The container with address 50.0.2.6 is running in a host in AWS with IP 54.193.189.77. It has addresses ["50.0.2.6","50.0.1.5"].

The Docker hosts (in the same AWS availability zone) can ping each other. Neither container can ping the other on the 50.0.2.x or 50.0.1.x addresses.

Clocker trying to stop already stopped docker daemon

I have 2 machines (using byon, trusty, docker 1.2) - when i create clocker cloud from scratch (both machines running docker daemon) everything works. Removing clocker cloud stops docker daemons.
Attempt to create clocker cloud again fails on stopping already stopped docker daemons.

From logs i suspect dockerhost status check returns 0 (docker stop/waiting in output) - it triggers dockerhost stop script execution failing with 1 exit status code.

0.8.2 cannot find conf/server-cert.pem

Downloaded v0.8.2 from clocker.io and ran ./bin/clocker.sh jclouds:softlayer:ams01. The installation of the DockerHosts failed with this exception:

2015-06-23 18:41:26,707 WARN  b.e.b.AbstractSoftwareProcessDriver [brooklyn-execmanager-uSWx77bc-85]: Error copying runtime resources
java.lang.RuntimeException: Error getting resource 'conf/server-cert.pem' for DockerHostImpl{id=AxGXNjjR}: java.io.IOException: 'conf/server-cert.pem' not found on classpath or filesystem
    at brooklyn.util.ResourceUtils.getResourceFromUrl(ResourceUtils.java:290) ~[brooklyn-core-0.7.0-20150603.164010-253.jar:0.7.0-SNAPSHOT]
    at brooklyn.location.basic.SshMachineLocation.installTo(SshMachineLocation.java:809) ~[brooklyn-core-0.7.0-20150603.164010-253.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.basic.AbstractSoftwareProcessSshDriver.copyResource(AbstractSoftwareProcessSshDriver.java:368) ~[brooklyn-software-base-0.7.0-20150603.164055-252.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.basic.AbstractSoftwareProcessDriver.copyResource(AbstractSoftwareProcessDriver.java:417) ~[brooklyn-software-base-0.7.0-20150603.164055-252.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.basic.AbstractSoftwareProcessDriver.copyRuntimeResources(AbstractSoftwareProcessDriver.java:346) ~[brooklyn-software-base-0.7.0-20150603.164055-252.jar:0.7.0-SNAPSHOT]
    at brooklyn.entity.basic.AbstractSoftwareProcessDriver$9.run(AbstractSoftwareProcessDriver.java:155) [brooklyn-software-base-0.7.0-20150603.164055-252.jar:0.7.0-SNAPSHOT]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_51]
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:337) [brooklyn-core-0.7.0-20150603.164010-253.jar:0.7.0-SNAPSHOT]
    at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469) [brooklyn-core-0.7.0-20150603.164010-253.jar:0.7.0-SNAPSHOT]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.io.IOException: 'conf/server-cert.pem' not found on classpath or filesystem
    at brooklyn.util.ResourceUtils.getResourceFromUrl(ResourceUtils.java:287) ~[brooklyn-core-0.7.0-20150603.164010-253.jar:0.7.0-SNAPSHOT]
    ... 12 common frames omitted

Entity shutdown leaves container running

Stopping an entity does not always stop the associated Docker container, although the DockerContainerLocation has been closed and the DockerContainer entity has had the stop effector called. This results in the host container scan picking up the orphaned containers and generating new DockerContainer entities.

SSH error when starting several entities

When starting several entities in containers created by Docker I regularly see one or more entities fail with SSH connection issues. I've given a full description in this Gist: https://gist.github.com/sjcorbett/11223441. To reproduce the pertinent comments:

The crucial log lines are:

2014-04-23 17:45:52,755 ERROR n.s.s.transport.TransportImpl [reader]: Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
2014-04-23 17:45:52,756 ERROR net.schmizz.concurrent.Promise [brooklyn-execmanager-Jprf8QYP-85]: <<chan#6 / close>> woke to: net.schmizz.sshj.connection.ConnectionException: Broken transport; encountered EOF

In an attempt to glean more information I added these lines to SshjTool before the return returnVal statement in acquire(C action):

                if (returnVal == null) {
                    String message = "SshAction "+action+" returned null.";
                    if (action instanceof ShellAction) {
                        ShellAction shellAction = (ShellAction) action;
                        Shell shell = shellAction.shell;
                        LOG.warn("** SshAction shell is: {}, class={}", shell, shell.getClass());
                        if (shell instanceof SessionChannel) {
                            SessionChannel channel = (SessionChannel) shell;
                            message = String.format("%s channelExitStatus=%d, channelExitErrorMessage=%s",
                                    message, channel.getExitStatus(), channel.getExitErrorMessage());
                        }
                    }
                    LOG.warn(message);
                }

However, with these lines in place the issue stopped occurring! Or at least, it didn't reoccur in fifteen-odd attempts. So it's a bit of a Heisenbug.

Integration with Docker Hub

Hello,

I have a question. At present, is possible to integrate docker containers from Docker Hub repository? I watched something about custom app deployment in brooklyn webpages

Install failures blocks startup of other entities in clocker containers

On failure of install for an entity (being started in a container), it seemingly blocked the startup of all other entities.

I tried to launch a MemSQL cluster but the first node failed to install (ignore what the error says; that would be a topic for a different bug report).

2014-09-09 11:09:03,189 INFO b.u.t.s.ProcessTaskWrapper [brooklyn-execmanager-ZJp0gyZe-443]: STDIN for problem in Task[ssh: initializing on-box base dir ./brooklyn-managed-processes:iyH9XHpw]:
( mkdir -p "${BASE_DIR}" || ( ( if test "$UID" -eq 0; then ( mkdir -p "${BASE_DIR}" ); else sudo -E -n -S -- mkdir -p "${BASE_DIR}"; fi ) && ( if test "$UID" -eq 0; then ( chown root "${BASE_DIR}" ); else sudo -E -n -S -- chown root "${BASE_DIR}"; fi ) ) )
cd ~
cd ${BASE_DIR}
echo BASE_DIR_RESULT':'pwd:BASE_DIR_RESULT
2014-09-09 11:09:03,189 DEBUG b.u.task.BasicExecutionManager [brooklyn-execmanager-ZJp0gyZe-443]: Exception running task Taskssh: initializing on-box base dir ./brooklyn-managed-processes:iyH9XHpw: SSH task ended with exit code 1 when 0 was required, in Task[ssh: initializing on-box base dir ./brooklyn-managed-processes:iyH9XHpw]: initializing on-box base dir ./brooklyn-managed-processes
java.lang.IllegalStateException: SSH task ended with exit code 1 when 0 was required, in Task[ssh: initializing on-box base dir ./brooklyn-managed-processes:iyH9XHpw]: initializing on-box base dir ./brooklyn-managed-processes
at brooklyn.util.task.system.ProcessTaskWrapper$ProcessTaskInternalJob.logWithDetailsAndThrow(ProcessTaskWrapper.java:138) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.system.ProcessTaskWrapper$ProcessTaskInternalJob.call(ProcessTaskWrapper.java:112) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [na:1.6.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [na:1.6.0_65]
at java.lang.Thread.run(Thread.java:695) [na:1.6.0_65]

The stacktrace showed DockerHostLocation waiting for the image:

java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <78ca1cc78> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
at brooklyn.location.docker.DockerHostLocation.waitForImage(DockerHostLocation.java:233)
at brooklyn.location.docker.DockerHostLocation.obtain(DockerHostLocation.java:166)
at brooklyn.location.docker.DockerLocation.obtain(DockerLocation.java:155)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5$1.call(MachineLifecycleEffectorTasks.java:261)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5$1.call(MachineLifecycleEffectorTasks.java:1)
at brooklyn.util.task.Tasks.withBlockingDetails(Tasks.java:93)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5.call(MachineLifecycleEffectorTasks.java:259)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5.call(MachineLifecycleEffectorTasks.java:1)
at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)

And all other nodes (not just those sharing the same image) waiting for the mutex before they could create a location:

"brooklyn-execmanager-ZJp0gyZe-148" daemon prio=5 tid=7fe31c505000 nid=0x11e590000 waiting on condition [11e58f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <78c919c58> (a java.util.concurrent.Semaphore$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
at brooklyn.util.mutex.SemaphoreWithOwners.acquire(SemaphoreWithOwners.java:51)
at brooklyn.util.mutex.MutexSupport.acquireMutex(MutexSupport.java:78)
at brooklyn.location.docker.DockerLocation.acquireMutex(DockerLocation.java:260)
at brooklyn.location.docker.DockerLocation.obtain(DockerLocation.java:122)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5$1.call(MachineLifecycleEffectorTasks.java:261)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5$1.call(MachineLifecycleEffectorTasks.java:1)
at brooklyn.util.task.Tasks.withBlockingDetails(Tasks.java:93)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5.call(MachineLifecycleEffectorTasks.java:259)
at brooklyn.entity.software.MachineLifecycleEffectorTasks$5.call(MachineLifecycleEffectorTasks.java:1)
at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:318)
at brooklyn.util.task.BasicExecutionManager$2.call(BasicExecutionManager.java:389)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)

Make Docker command and service execution configurable

Some time ago Docker renamed the command line executable and the service from docker to docker.io (at least in the Ubuntu packages from Docker) and therefore Clocker can neither execute the Docker executable nor control the service. For ease of use many users create a symbolic link docker but that doesn't solve the service problem. As of Docker 0.5.3 the Docker installer creates a unix group docker to enable normal users to run docker without super user rights and I think this is the preferable way. All this is documented in the Docker documentation, so I think many users are using Docker this way.

So I think a way out could be to make the name of the executable and the service configurable as well as the use of sudo for both. I think many users don't like to have a sudoer without a password on their system as this is a security issue.

Docker cloud start blocks forever when etcd node fails to start

I ran Clocker with Calico, version 1.1.0.PREVIEW_20150904.

One of the Etcd nodes failed to start. Brooklyn reported "No information on whether this service is running" and no activities.

This caused one of the Docker hosts to block forever in its postStart method:

In progress - Waiting for attributeSupplierWhenReady (blocked on docker-2 waiting on sdn.agent.entity)

Entities.attributeSupplierWhenReady does not provide a clean way to give a time out.

Exception provisioning containers: SSLHandshakeException: Received fatal alert: bad_certificate bad_certificate

Built master/HEAD. Ran ./bin/clocker.sh jclouds:softlayer:ams01. Attempted to deploy riak-counters example (https://raw.githubusercontent.com/csabapalfi/riak-counters-example/master/clocker.yml). Every entity in the Riak application failed at provisioning with this exception:

2015-07-02 12:19:12,861 WARN  b.l.jclouds.JcloudsLocation [brooklyn-execmanager-ZIBiVbK0-256]: IOException found...
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.7.0_51]
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[na:1.7.0_51]
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1959) ~[na:1.7.0_51]
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077) ~[na:1.7.0_51]
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) ~[na:1.7.0_51]
    ....

The DockerContainer children of the host entities are stuck spinning as if they're starting, but the start task failed with:

Failed after 3.02s: Error invoking start at DockerContainerImpl{id=PUDU0uTO}: Unable to access cloud JcloudsLocation[docker:https://81.95.144.60:2376:conf/server-cert.pem/docker:https://81.95.144.60:2376@zW8EqnJY] to resolve PortableTemplateBuilder[ports=[22, 8093, 7999, 8099, 8098, 6000, 8087, 8985, 4369], imageId=bacb06d7f62997716eae9d7fb8f5bf925ab6dc6efc000d38edc1214ba5f43f8e, hardwareId=small, imageChooserFunction=brooklyn.location.jclouds.BrooklynImageChooser$ImageChooserFromOrdering@78c9cffa]: org.jclouds.http.HttpResponseException: Received fatal alert: bad_certificate connecting to GET https://81.95.144.60:2376/v1.16/images/json HTTP/1.1

brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking start at DockerContainerImpl{id=PUDU0uTO}: Unable to access cloud JcloudsLocation[docker:https://81.95.144.60:2376:conf/server-cert.pem/docker:https://81.95.144.60:2376@zW8EqnJY] to resolve PortableTemplateBuilder[ports=[22, 8093, 7999, 8099, 8098, 6000, 8087, 8985, 4369], imageId=bacb06d7f62997716eae9d7fb8f5bf925ab6dc6efc000d38edc1214ba5f43f8e, hardwareId=small, imageChooserFunction=brooklyn.location.jclouds.BrooklynImageChooser$ImageChooserFromOrdering@78c9cffa]: org.jclouds.http.HttpResponseException: Received fatal alert: bad_certificate connecting to GET https://81.95.144.60:2376/v1.16/images/json HTTP/1.1
    at brooklyn.management.internal.EffectorUtils.handleEffectorException(EffectorUtils.java:270)
    at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:255)
    at brooklyn.entity.basic.MethodEffector.call(MethodEffector.java:149)
    at brooklyn.entity.trait.Startable$StartEffectorBody.call(Startable.java:57)
    at brooklyn.entity.trait.Startable$StartEffectorBody.call(Startable.java:51)
    at brooklyn.entity.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:83)
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:337)
    at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Unable to access cloud JcloudsLocation[docker:https://81.95.144.60:2376:conf/server-cert.pem/docker:https://81.95.144.60:2376@zW8EqnJY] to resolve PortableTemplateBuilder[ports=[22, 8093, 7999, 8099, 8098, 6000, 8087, 8985, 4369], imageId=bacb06d7f62997716eae9d7fb8f5bf925ab6dc6efc000d38edc1214ba5f43f8e, hardwareId=small, imageChooserFunction=brooklyn.location.jclouds.BrooklynImageChooser$ImageChooserFromOrdering@78c9cffa]: org.jclouds.http.HttpResponseException: Received fatal alert: bad_certificate connecting to GET https://81.95.144.60:2376/v1.16/images/json HTTP/1.1
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:314)
    at brooklyn.management.internal.EffectorUtils.invokeMethodEffector(EffectorUtils.java:250)
    ... 10 more
Caused by: java.lang.IllegalStateException: Unable to access cloud JcloudsLocation[docker:https://81.95.144.60:2376:conf/server-cert.pem/docker:https://81.95.144.60:2376@zW8EqnJY] to resolve PortableTemplateBuilder[ports=[22, 8093, 7999, 8099, 8098, 6000, 8087, 8985, 4369], imageId=bacb06d7f62997716eae9d7fb8f5bf925ab6dc6efc000d38edc1214ba5f43f8e, hardwareId=small, imageChooserFunction=brooklyn.location.jclouds.BrooklynImageChooser$ImageChooserFromOrdering@78c9cffa]: org.jclouds.http.HttpResponseException: Received fatal alert: bad_certificate connecting to GET https://81.95.144.60:2376/v1.16/images/json HTTP/1.1
    at brooklyn.location.jclouds.JcloudsLocation.buildTemplate(JcloudsLocation.java:1420)
    at brooklyn.location.jclouds.JcloudsLocation.obtainOnce(JcloudsLocation.java:660)
    at brooklyn.location.jclouds.JcloudsLocation.obtain(JcloudsLocation.java:596)
    at brooklyn.entity.container.docker.DockerContainerImpl.createLocation(DockerContainerImpl.java:453)
    at brooklyn.entity.container.docker.DockerContainerImpl.start(DockerContainerImpl.java:570)
    at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1085)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:909)
    at groovy.lang.DelegatingMetaClass.invokeMethod(DelegatingMetaClass.java:149)
    at groovy.lang.MetaObjectProtocol$invokeMethod.call(Unknown Source)
    at brooklyn.util.GroovyJavaMethods.invokeMethodOnMetaClass(GroovyJavaMethods.groovy:144)
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodLocal(AbstractManagementContext.java:286)
    at brooklyn.management.internal.AbstractManagementContext.invokeEffectorMethodSync(AbstractManagementContext.java:310)
    ... 11 more
Caused by: org.jclouds.http.HttpResponseException: Received fatal alert: bad_certificate connecting to GET https://81.95.144.60:2376/v1.16/images/json HTTP/1.1
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:117)
    at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
    at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
    at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
    at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
    at com.sun.proxy.$Proxy193.listImages(Unknown Source)
    at org.jclouds.docker.compute.strategy.DockerComputeServiceAdapter.listImages(DockerComputeServiceAdapter.java:183)
    at org.jclouds.docker.compute.strategy.DockerComputeServiceAdapter.listImages(DockerComputeServiceAdapter.java:63)
    at org.jclouds.compute.config.ComputeServiceAdapterContextModule$2.get(ComputeServiceAdapterContextModule.java:121)
    at org.jclouds.compute.config.ComputeServiceAdapterContextModule$2.get(ComputeServiceAdapterContextModule.java:118)
    at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:73)
    at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:57)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
    at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:119)
    at org.jclouds.compute.suppliers.ImageCacheSupplier.get(ImageCacheSupplier.java:64)
    at org.jclouds.compute.domain.internal.TemplateBuilderImpl.getImages(TemplateBuilderImpl.java:881)
    at org.jclouds.compute.domain.internal.TemplateBuilderImpl.build(TemplateBuilderImpl.java:681)
    at brooklyn.location.jclouds.templates.PortableTemplateBuilder.newJcloudsTemplate(PortableTemplateBuilder.java:70)
    at brooklyn.location.jclouds.templates.PortableTemplateBuilder.build(PortableTemplateBuilder.java:48)
    at brooklyn.location.jclouds.JcloudsLocation.buildTemplate(JcloudsLocation.java:1398)
    ... 27 more
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1959)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1077)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
    at com.squareup.okhttp.Connection.upgradeToTls(Connection.java:241)
    at com.squareup.okhttp.Connection.connect(Connection.java:158)
    at com.squareup.okhttp.Connection.connectAndSetOwner(Connection.java:174)
    at com.squareup.okhttp.OkHttpClient$1.connectAndSetOwner(OkHttpClient.java:120)
    at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:131)
    at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:312)
    at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:235)
    at com.squareup.okhttp.Call.getResponse(Call.java:262)
    at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:219)
    at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:192)
    at com.squareup.okhttp.Call.execute(Call.java:79)
    at org.jclouds.http.okhttp.OkHttpCommandExecutorService.invoke(OkHttpCommandExecutorService.java:145)
    at org.jclouds.http.okhttp.OkHttpCommandExecutorService.invoke(OkHttpCommandExecutorService.java:58)
    at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:96)
    ... 53 more

The Weave containers in each of the Docker hosts appear to be running normally.

Errors resolving bundles

Adding the entities in https://github.com/brooklyncentral/clocker/blob/001d8a059e9cebb1bb94273fd5e4506cd7ff21bb/clocker.bom to a catalogue fails with:

Unresolved constraint in bundle io.brooklyn.clocker.brooklyn-clocker-patches [134]: Unable to resolve 134.0: missing requirement [134.0] osgi.wiring.package; (osgi.wiring.package=javax.inject)

Unresolved constraint in bundle io.brooklyn.networking.brooklyn-networking-cloudstack [140]: Unable to resolve 140.0: missing requirement [140.0] osgi.wiring.package; (osgi.wiring.package=org.apache.http)

I fixed a couple of straightforward ones in #157.

Clocker install fails with exit code -1

When trying to start Clocker on a vcloud-director private cloud, the install failed with exit code -1 (which almost certainly means the ssh connection being dropped).

Is there a different way we should execute these long-running ssh commands, so that we don't get this problem?

    2014-11-24 14:48:07,041 WARN  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-OLAp73yN-235]: Execution failed, invalid result -1 for installing DockerHostImpl{id=N1W0vyeI} (throwing)
    2014-11-24 14:48:07,041 INFO  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-OLAp73yN-235]: STDOUT of problem in Task[ssh: installing DockerHostImpl{id=N1W0vyeI}:ApBYvr7Y]:
    /usr/bin/wget
    2014-11-24 14:48:07,041 INFO  b.e.b.lifecycle.ScriptHelper [brooklyn-execmanager-OLAp73yN-235]: STDIN of problem in Task[ssh: installing DockerHostImpl{id=N1W0vyeI}:ApBYvr7Y]:
    export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/DockerHost_1.2"
    mkdir -p $INSTALL_DIR
    cd $INSTALL_DIR
    test -f BROOKLYN && exit 0
    { { test -z `which yum` && true ; } || { { which wget || ( { which zypper && { echo zypper exists, doing refresh && (( if test "$UID" -eq 0; then ( zypper --non-interactive --no-gpg-checks refresh ); else sudo -E -n -S -- zypper --non-interactive --no-gpg-checks refresh; fi ) || true) && ( if test "$UID" -eq 0; then ( zypper --non-interactive --no-gpg-checks install wget ); else sudo -E -n -S -- zypper --non-interactive --no-gpg-checks install wget; fi ) ; } ; } || { which apt-get && { echo apt-get exists, doing update && export DEBIAN_FRONTEND=noninteractive && (( if test "$UID" -eq 0; then ( apt-get update ); else sudo -E -n -S -- apt-get update; fi ) || true) && ( if test "$UID" -eq 0; then ( apt-get install -y --allow-unauthenticated wget ); else sudo -E -n -S -- apt-get install -y --allow-unauthenticated wget; fi ) ; } ; } || { which yum && { echo yum exists, doing update && (( if test "$UID" -eq 0; then ( yum check-update ); else sudo -E -n -S -- yum check-update; fi ) || true) && ( if test "$UID" -eq 0; then ( yum -y --nogpgcheck install wget ); else sudo -E -n -S -- yum -y --nogpgcheck install wget; fi ) ; } ; } || { which brew && brew install wget ; } || { which port && ( if test "$UID" -eq 0; then ( port install wget ); else sudo -E -n -S -- port install wget; fi ) ; } || (( echo "WARNING: no known/successful package manager to install wget, may fail subsequently" | tee /dev/stderr ) || true) ) ; } && ( ( if test "$UID" -eq 0; then ( rpm -qa | grep epel-release ); else sudo -E -n -S -- rpm -qa | grep epel-release; fi ) || ( if test "$UID" -eq 0; then ( rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm ); else sudo -E -n -S -- rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm; fi ) ) ; } ; }
    ( { which yum && { echo yum exists, doing update && (( if test "$UID" -eq 0; then ( yum check-update ); else sudo -E -n -S -- yum check-update; fi ) || true) && ( if test "$UID" -eq 0; then ( yum -y --nogpgcheck install docker-io ); else sudo -E -n -S -- yum -y --nogpgcheck install docker-io; fi ) ; } ; } || (( echo "WARNING: no known/successful package manager to install {yum=docker-io}, may fail subsequently" | tee /dev/stderr ) || true) )
    date > $INSTALL_DIR/BROOKLYN
    2014-11-24 14:48:07,042 DEBUG b.u.task.BasicExecutionManager [brooklyn-execmanager-OLAp73yN-235]: Exception running task Task[ssh: installing DockerHostImpl{id=N1W0vyeI}:ApBYvr7Y] (rethrowing): Execution failed, invalid result -1 for installing DockerHostImpl{id=N1W0vyeI} 

The log also shows the following, but it's hard to tell if it was definitely associated with the docker command rather than another app:

    2014-11-24 14:48:07,037 ERROR n.s.s.transport.TransportImpl [reader]: Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF
    2014-11-24 14:48:07,039 ERROR n.s.s.transport.TransportImpl [reader]: Dying because - net.schmizz.sshj.transport.TransportException: Broken transport; encountered EOF 

Clocker stop failed

When stopping the application described in #150.

An invocation of the Stop effector on the Docker Cloud application failed with root cause a failure to stop one of the EtcdNodes (which had failed earlier in the deployment).

org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at BasicApplicationImpl{id=TaYgZIK2}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at DockerInfrastructureImpl{id=xJiaRrBc}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at CalicoNetworkImpl{id=pGSGOteZ}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at EtcdClusterImpl{id=Gk58FaAu}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: 1 of 3 parallel child tasks failed: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking stop at EtcdNodeImpl{id=f71iRD6T}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: java.lang.IllegalStateException: Execution failed, invalid result 1 for removeMember EtcdNodeImpl{id=f71iRD6T}

The exact problem was:

Failed after 9.76s: Error invoking stop at EtcdNodeImpl{id=f71iRD6T}: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: java.lang.IllegalStateException: Execution failed, invalid result 1 for removeMember EtcdNodeImpl{id=f71iRD6T}

Caused by: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: java.lang.IllegalStateException: Execution failed, invalid result 1 for removeMember EtcdNodeImpl{id=f71iRD6T}
    at org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:102)
    at org.apache.brooklyn.util.core.task.BasicTask.getUnchecked(BasicTask.java:353)
    at org.apache.brooklyn.entity.software.base.lifecycle.ScriptHelper.execute(ScriptHelper.java:337)
    at brooklyn.entity.nosql.etcd.EtcdNodeSshDriver.leaveCluster(EtcdNodeSshDriver.java:221)
    at brooklyn.entity.nosql.etcd.EtcdNodeSshDriver.stop(EtcdNodeSshDriver.java:151)
    at org.apache.brooklyn.entity.software.base.SoftwareProcessDriverLifecycleEffectorTasks.stopProcessesAtMachine(SoftwareProcessDriverLifecycleEffectorTasks.java:232)
    at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopProcessesAtMachineTask.call(MachineLifecycleEffectorTasks.java:771)
    at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$StopProcessesAtMachineTask.call(MachineLifecycleEffectorTasks.java:768)
    ... 6 more

I had to get into the debug log to find the output from the stop commands:

2015-09-08 14:52:25,158 WARN  o.a.b.e.s.b.l.ScriptHelper [brooklyn-execmanager-YbH392t6-5227]: Execution failed, invalid result 1 for removeMember EtcdNodeImpl{id=f71iRD6T} (throwing)
2015-09-08 14:52:25,159 INFO  o.a.b.e.s.b.l.ScriptHelper [brooklyn-execmanager-YbH392t6-5227]: STDERR of problem in Task[ssh: removeMember EtcdNodeImpl{id=f71iRD6T}:uML2O0pH]:
Recieved an error trying to remove member 8e21d999ac31510d: context deadline exceeded

2015-09-08 14:52:25,159 INFO  o.a.b.e.s.b.l.ScriptHelper [brooklyn-execmanager-YbH392t6-5227]: STDOUT of problem in Task[ssh: removeMember EtcdNodeImpl{id=f71iRD6T}:uML2O0pH]:
Executed /tmp/brooklyn-20150908-145217770-pIs7-removeMember_EtcdNodeImpl_id_f.sh, result 1

2015-09-08 14:52:25,159 INFO  o.a.b.e.s.b.l.ScriptHelper [brooklyn-execmanager-YbH392t6-5227]: STDIN of problem in Task[ssh: removeMember EtcdNodeImpl{id=f71iRD6T}:uML2O0pH]:
cd /home/users/brooklyn/brooklyn-managed-processes/apps/TaYgZIK2/entities/EtcdNode_f71iRD6T
/home/users/brooklyn/brooklyn-managed-processes/installs/EtcdNode_2.0.11/etcd-v2.0.11-linux-amd64/etcdctl --peers http://10.235.43.152:2379 member remove 8e21d999ac31510d

DockerHosts were stopped as expected.

Timed in `pre-start` running `docker inspect` when deploying riak cluster

When starting a Riak cluster in a low-powered Clocker environment (two hosts with 3G each), the pre-start timed out with the exception shown below.

The other 4 out of 5 riak entities started successfully.

The 3 jboss app-server entities all failed with a similar error.

Running the command on the docker host now, it returns fast. Looking at docker ps -a, I see the container listed fine.

Is there something wrong with our timeouts? Why does it say it failed after 3m 18s when the ssh command completed at about that time with exit code 0?

Failed after 3m 18s: Error invoking start at RiakNodeImpl{id=E6iaa3Cu}: Timed out running command: ( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )

brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking start at RiakNodeImpl{id=E6iaa3Cu}: Timed out running command: ( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )
    at brooklyn.management.internal.EffectorUtils.handleEffectorException(EffectorUtils.java:263)
    at brooklyn.entity.effector.EffectorTasks$EffectorBodyTaskFactory$2.handleException(EffectorTasks.java:90)
    at brooklyn.util.task.DynamicSequentialTask.handleException(DynamicSequentialTask.java:447)
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:395)
    at brooklyn.util.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:469)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException: Timed out running command: ( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:91)
    at brooklyn.util.task.BasicTask.getUnchecked(BasicTask.java:354)
    at brooklyn.util.task.DynamicSequentialTask.drain(DynamicSequentialTask.java:477)
    at brooklyn.util.task.DynamicTasks.drain(DynamicTasks.java:314)
    at brooklyn.util.task.DynamicTasks.waitForLast(DynamicTasks.java:303)
    at brooklyn.entity.software.MachineLifecycleEffectorTasks.start(MachineLifecycleEffectorTasks.java:228)
    at brooklyn.entity.software.MachineLifecycleEffectorTasks$1.call(MachineLifecycleEffectorTasks.java:165)
    at brooklyn.entity.software.MachineLifecycleEffectorTasks$1.call(MachineLifecycleEffectorTasks.java:1)
    at brooklyn.entity.effector.EffectorTasks$EffectorBodyTaskFactory$1.call(EffectorTasks.java:82)
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:337)
    ... 5 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Timed out running command: ( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )
    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:188)
    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63)
    at brooklyn.util.task.BasicTask.get(BasicTask.java:343)
    at brooklyn.util.task.BasicTask.getUnchecked(BasicTask.java:352)
    ... 13 more
Caused by: java.lang.IllegalStateException: Timed out running command: ( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )
    at brooklyn.entity.machine.MachineEntityImpl.execCommandTimeout(MachineEntityImpl.java:175)
    at brooklyn.entity.container.docker.DockerHostImpl.runDockerCommandTimeout(DockerHostImpl.java:325)
    at brooklyn.entity.container.docker.DockerHostImpl.runDockerCommand(DockerHostImpl.java:318)
    at brooklyn.location.docker.DockerContainerLocation.getSubnetIp(DockerContainerLocation.java:259)
    at brooklyn.location.docker.DockerContainerLocation.getSubnetHostname(DockerContainerLocation.java:254)
    at brooklyn.location.basic.Machines.getSubnetHostname(Machines.java:43)
    at brooklyn.entity.software.MachineLifecycleEffectorTasks$6.run(MachineLifecycleEffectorTasks.java:324)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    ... 6 more

Drilling into the ssh task via the web-console, it shows Completed after 3m 11s with stdin:

( if test "$UID" -eq 0; then ( docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408 ); else sudo -E -n -S -- docker inspect --format={{.NetworkSettings.IPAddress}} 07f47ec4ef71a4c6dc4667b682f30515c742e7974b3978907a6884b7fbe5c408; fi )

And stdout 172.17.0.19. There was no stderr or env.

Jclouds Driver Enhancements

Add options for CPU shares and other configuration enhancements available in latest version of the jclouds-docker driver at jclouds/jclouds-labs#57

SameServerEntity ports not mapped

Try to deploy this app to a Docker container:

name: App
location: my-docker-cloud
services:
- type: brooklyn.entity.basic.SameServerEntity
  name: Redises
  brooklyn.children:
  - type: brooklyn.entity.nosql.redis.RedisStore

I observe this NullPointerException:

2014-06-19 12:29:23,031 DEBUG b.location.jclouds.JcloudsUtil [brooklyn-execmanager-LPBGewXV-121]: Docker will forward these ports {22/tcp=[{HostIp=0.0.0.0, HostPort=49156}]}
2014-06-19 12:29:23,031 DEBUG b.l.d.DockerContainerLocation [brooklyn-execmanager-LPBGewXV-121]: Docker mapped port 6379 to null for Container 757d1a8a5c7d40cd3397fdfb6ba074d6dcf6c7ebe5d5efc9c8df06db2031364b
2014-06-19 12:29:23,032 DEBUG b.u.task.BasicExecutionManager [brooklyn-execmanager-LPBGewXV-121]: Exception running task Task[pre-start [SUB-TASK, Wrapped[contextEntity:RedisStoreImpl{id=l6T3DxDI}]]; e8wKFo44] (rethrowing): Cannot convert config value 6379-65535 for sensor Sensor: redis.port (java.lang.Integer): java.lang.NullPointerException
java.lang.IllegalArgumentException: Cannot convert config value 6379-65535 for sensor Sensor: redis.port (java.lang.Integer): java.lang.NullPointerException
    at brooklyn.event.basic.AttributeSensorAndConfigKey.getAsSensorValue(AttributeSensorAndConfigKey.java:95) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.event.feed.ConfigToAttributes.apply(ConfigToAttributes.java:28) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.event.feed.ConfigToAttributes.apply(ConfigToAttributes.java:17) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.entity.software.MachineLifecycleEffectorTasks.preStartCustom(MachineLifecycleEffectorTasks.java:349) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.entity.basic.SoftwareProcessDriverLifecycleEffectorTasks.preStartCustom(SoftwareProcessDriverLifecycleEffectorTasks.java:69) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.entity.software.MachineLifecycleEffectorTasks$6.run(MachineLifecycleEffectorTasks.java:289) ~[brooklyn-software-base-0.7.0-SNAPSHOT.jar:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_51]
    at brooklyn.util.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:296) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.util.task.BasicExecutionManager$3.call(BasicExecutionManager.java:353) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.lang.NullPointerException: null
    at brooklyn.location.docker.DockerContainerLocation.getMappedPort(DockerContainerLocation.java:111) ~[brooklyn-clocker-docker-0.5.0-SNAPSHOT.jar:na]
    at brooklyn.location.docker.DockerContainerLocation.obtainPort(DockerContainerLocation.java:128) ~[brooklyn-clocker-docker-0.5.0-SNAPSHOT.jar:na]
    at brooklyn.event.basic.PortAttributeSensorAndConfigKey.convertConfigToSensor(PortAttributeSensorAndConfigKey.java:78) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.event.basic.PortAttributeSensorAndConfigKey.convertConfigToSensor(PortAttributeSensorAndConfigKey.java:1) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    at brooklyn.event.basic.AttributeSensorAndConfigKey.getAsSensorValue(AttributeSensorAndConfigKey.java:93) ~[brooklyn-core-0.7.0-SNAPSHOT.jar:na]
    ... 12 common frames omitted

Utilise spotify docker client

Spotify released a Java docker-client some time ago now: https://github.com/spotify/docker-client

I think it would be awesome if we can switch to that instead of using the shell to execute docker commands.

It's just an idea at this point but I have high hopes for that making our lives simpler.

Error bringing up 2 node docker cluster on EC2, calico node error

Hi
Been getting this error when trying to spin up a cluster in ec2, I've gotten a similar error when I select Weave. I've been using the latest distro available under releases

2015-08-25 08:41:52,263 INFO  Adding first node EtcdClusterImpl{id=Fsa4HRJC}: EtcdNodeImpl{id=hvTtRCqC}; calico1 to cluster
2015-08-25 08:41:54,663 INFO  Adding first node EtcdClusterImpl{id=Fsa4HRJC}: EtcdNodeImpl{id=hvTtRCqC}; calico1 to cluster
2015-08-25 08:41:54,751 INFO  Starting CalicoNodeImpl{id=MCZP7WJx} on machine SshMachineLocation[ec2-54-196-17-21.compute-1.amazonaws.com:ec2-54-196-17-21.compute-1.amazonaws.com/54.196.17.21:22@MPQ0lxj1]
2015-08-25 08:41:57,875 WARN  Configuration being made to SshMachineLocation[ec2-54-196-17-21.compute-1.amazonaws.com:ec2-54-196-17-21.compute-1.amazonaws.com/54.196.17.21:22@MPQ0lxj1] after deployment; may not be supported in future versions
2015-08-25 08:42:07,175 INFO  Launching first calico service at 54.196.17.21
2015-08-25 08:42:07,995 WARN  Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx} (throwing)
2015-08-25 08:42:07,995 INFO  STDERR of problem in Task[ssh: launching CalicoNodeImpl{id=MCZP7WJx}:Y9JX3jEB]:
WARNING: ipv6 forwarding is not enabled.

2015-08-25 08:42:07,995 INFO  STDOUT of problem in Task[ssh: launching CalicoNodeImpl{id=MCZP7WJx}:Y9JX3jEB]:
Unable to run docker commands. Is the docker daemon running?
Executed /tmp/brooklyn-20150825-084207177-DVx7-launching_CalicoNodeImpl_id_MC.sh, result 1

2015-08-25 08:42:07,995 INFO  STDIN of problem in Task[ssh: launching CalicoNodeImpl{id=MCZP7WJx}:Y9JX3jEB]:
export RUN_DIR="/home/users/bitsofinfo.dog/brooklyn-managed-processes/apps/YhD6Amqj/entities/CalicoNode_MCZP7WJx"
mkdir -p $RUN_DIR
cd $RUN_DIR
( if test "$UID" -eq 0; then ( /home/users/bitsofinfo.dog/brooklyn-managed-processes/installs/CalicoNode_0.4.8/calicoctl node --ip=54.196.17.21 ); else sudo -E -n -S -- /home/users/bitsofinfo.dog/brooklyn-managed-processes/installs/CalicoNode_0.4.8/calicoctl node --ip=54.196.17.21; fi )
2015-08-25 08:42:08,001 WARN  Setting BasicGroupImpl{id=lIhe6lD5} on-fire due to problems when expected null / up=true: {service-lifecycle-indicators-from-children-and-members=Required entity not healthy: CalicoNodeImpl{id=MCZP7WJx}}
2015-08-25 08:42:08,001 WARN  Error invoking start at CalicoNodeImpl{id=MCZP7WJx}: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
2015-08-25 08:42:08,006 WARN  Error processing subscriptions to LSM.publish(DockerHostImpl{id=HJIQXtop}.Sensor: service.isUp (java.lang.Boolean)=true @ 1440506507995): brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking start at CalicoNodeImpl{id=MCZP7WJx}: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
brooklyn.util.exceptions.PropagatedRuntimeException:
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:97) ~[brooklyn-utils-common-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking start at CalicoNodeImpl{id=MCZP7WJx}: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_45]
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException: Error invoking start at CalicoNodeImpl{id=MCZP7WJx}
    at org.apache.brooklyn.core.management.internal.EffectorUtils.handleEffectorException(EffectorUtils.java:270) ~[brooklyn-core-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException:
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:97) ~[brooklyn-utils-common-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: brooklyn.util.exceptions.PropagatedRuntimeException: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_45]
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException:
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:97) ~[brooklyn-utils-common-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: brooklyn.util.exceptions.PropagatedRuntimeException: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_45]
Caused by: brooklyn.util.exceptions.PropagatedRuntimeException:
    at brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:97) ~[brooklyn-utils-common-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) [na:1.8.0_45]
Caused by: java.lang.IllegalStateException: Execution failed, invalid result 1 for launching CalicoNodeImpl{id=MCZP7WJx}
    at brooklyn.entity.basic.lifecycle.ScriptHelper.logWithDetailsAndThrow(ScriptHelper.java:389) ~[brooklyn-software-base-0.8.0-SNAPSHOT.jar:0.8.0-SNAPSHOT]
2015-08-25 08:42:50,324 WARN  Task[post-launch:R5cZnevq] was never submitted; did the code create it and forget to run it? ('cancel' the task to suppress this message)
2015-08-25 08:42:50,324 WARN  Task[post-start:QkNSHb79] was never submitted; did the code create it and forget to run it? ('cancel' the task to suppress this message)
2015-08-25 08:56:43,841 WARN  Software process entity DockerHostImpl{id=kcC53vSs} did not pass is-running check within the required 15m limit (15m 1s elapsed) (throwing)
2015-08-25 08:56:44,045 WARN  Service is not up when setting running on DockerHostImpl{id=kcC53vSs}; delayed 203ms but Sensor: service.isUp (java.lang.Boolean) did not recover from false; not-up-indicators={service.process.isRunning=No information on whether this service is running}
2015-08-25 08:56:44,045 WARN  Setting DockerHostImpl{id=kcC53vSs} on-fire due to problems when expected running, up=false, not-up-indicators: {service.process.isRunning=No information on whether this service is running}
2015-08-25 08:56:44,045 WARN  Setting DockerHostImpl{id=kcC53vSs} on-fire due to problems when expected running, up=false, not-up-indicators: {service.process.isRunning=No information on whether this service is running}
2015-08-25 08:56:44,047 WARN  Error invoking start at DockerHostImpl{id=kcC53vSs}: Software process entity DockerHostImpl{id=kcC53vSs} did not pass is-running check within the required 15m limit (15m 1s elapsed)

Docker host setup with weave fails when Softlayer VMs get created in different VLANs

weave launch command fails as it can't connect to the other host and gets stuck:

Allocator range [169.254.0.0-169.255.0.0) awaiting consensus: Nodes known: 1, Quorum size: 2
Pending requests:
  Allocate for 64bb7a55698574e03bd3c5c6fe520dcaaf6a7a8a65f6939a06a5188d1793099e
Allocator default subnet: 169.254.0.0/16

Private key not used for ssh to docker host

With this brooklyn.properties

brooklyn.webconsole.security.users=admin
brooklyn.webconsole.security.user.admin.salt=oEY4
brooklyn.webconsole.security.user.admin.sha256=xyz

brooklyn.location.jclouds.aws-ec2.identity=xyz
brooklyn.location.jclouds.aws-ec2.credential=xyz
brooklyn.location.jclouds.aws-ec2.endpoint=https://ec2.eu-west-1.amazonaws.com

brooklyn.location.jclouds.aws-ec2.keyPair=brooklynkeypair
brooklyn.location.jclouds.aws-ec2.loginUser=ubuntu
brooklyn.location.jclouds.aws-ec2.loginUser.privateKeyFile=/.ssh/brooklynkeypair_rsa
brooklyn.location.jclouds.aws-ec2.loginUser.publicKeyFile=/.ssh/brooklynkeypair_rsa.pub
brooklyn.location.jclouds.aws-ec2.securityGroups=default
brooklyn.location.jclouds.aws-ec2.auto-generate-keypairs=false

I receive the following in the jclouds.log

2015-04-21 16:10:17,117 DEBUG o.j.h.h.BackoffLimitedRetryHandler [brooklyn-execmanager-CeJtzR6T-12]: Retry 49/50: delaying for 2000 ms: (ubuntu:pw[a59947d97d2d99beb2113b79bdae4813]@52.17.211.218:22) error acquiring {hostAndPort=52.17.212.228:22, loginUser=ubuntu, ssh=null, connectTimeout=60000, sessionTimeout=60000}: Exhausted available authentication methods

I have tested ssh from terminal on the machine running clocker/brooklyn, there is no pass phrase on the private key.

brooklyn.location.jclouds.aws-ec2.keyPair=brooklynkeypair is the keypair name from AWS.

VanillaDockerApplication Improvements

• Add configuration for ports and environment
• Create Dockerfile template that uses vanillaSoftwareProcess configuratoion
• Allow configuration with existing Dockerfile specified
• Create some example YAML files

Please update to use Brooklyn v0.7.0-M1

does it make sense to update?