If you leave etcd running it will build up a catalog of services from every bootstrap that has been run in the past. Since it's likely we will be provided etcd in an environment, we don't want bootstrap to only pull from what's currently in etcd, but to update etcd to the latest set of images in the docker registry.

The broker's build/Dockerfile should be updated to follow best practices employed by the apb execution environment base. If run without anyuid, the broker container will enter a crash loop due to the entrypoint.sh being unable to write to sed temp files in /etc/ansible-service-broker.

Container should be able to run as anyuid.

The broker's level of privileges is dependent on the level of privileges required by an APB.
Broker's Maximum Privilages = APB's Maximum Privilages
Let's define the the maximum privileges an APB should have and see if it can be lowered below cluster-admin.

On previous projects we used tito to build the rpms from the source tree. tito would tag the source tree and them update the version of the spec file and generate changelogs as well. This made it super easy to determine what software was included in a particular rpm.

We're not currently shipping rpms but we are building images. I'd like the image building process to tag the source tree. Something like this:

ansible-service-broker-VERSION

And when we tag images we should use that VERSION as well. That way when you go to our organization you can see that a particular version was built. We could probably build it into a tool (that's how tito was born too :)

Logs incorrectly state the broker is listening on localhost:1338, this could steer people in the wrong direction and should get updated.

The scripts/travis.sh script will eat linter errors. We should probably output the errors so that we can see them in the logs when it fails. Also print a suggestion to the user that they should run:

gofmt -d ./cmd
gofmt -d ./pkg

Header API versioning should be turned back on when k8s incubator issue is settled. Router was previously restricted to:

root := h.router.Headers("X-Broker-API-Version", "2.9").Subrouter()

Instead of using the client, let's create a go-client object that can connect with the Cluster.

https://github.com/fusor/ansible-service-broker/blob/master/pkg/broker/util.go#L13

The service broker logs to /tmp/ansible-service-broker.log, this file should be located in /var/log. Also need to configure a logrotate.

Most of the getters/setters in the dao are doing the same thing varying by type. Would be really nice to introduce some kind of "serializable" interface and do the work in one place. The gopher stole my generics.

Using jkig docker hub we had 0 images, broker failed to bootstrap.

Change Deprovision in broker.go to return a NotFound error instead of generic etcd error. The handler.go file seems to look at the error to see if it IsNotFound but that doesn't seem to be an http.StatusNotFound. We either update handler.go to use http codes or use the k8s codes.

Proposal: Multiple Registry’s

Goal: To allow users to provide multiple registries from the config. We should pull APB’s from all of the registries configured

Trello Card: https://trello.com/c/1KqTacLK/304-8-multiple-registry-adapters-configured-in-one-broker-instance

Changes to be made:

• app.App will be changed to have a slice of registries
• broker.Broker will be changed to have a slice of registries
• app.Config will contain a list of Registry Configs
• Bootstrap will loop through each configured registry and Load Specs
• Change registry config to have a Type and Name therefore users can name their registries.
• Combine logic of registries to simplify workflow - #180

Outstanding Questions

Questions with my thoughts underneath. I am looking for feedback on the options proposed.

1. How to deal with apb id collision

• Easiest solution is to error out and blow up. I don’t think that an ID collision should cause the broker to die
• Another idea is to warn if this occurs and take the one that is being entered.
• Redesign the data storage, so that registry -> spec. This will change the ID’s of everything away from UUID OR we create a mapping in etcd

2. What happens if a registry fails to load.

• Initial thought is to error out.
• We can add a configuration that will allow an person to determine if a registry should fail the broker or not.
• We can continue and warn if and only error if all registries have an error.

3. Should loading of specs be parallelised

• Initial thought is simpler is easier to understand, we can go back and add goroutines at a later time.
• If this is the case then we need to make sure 2 makes sense.

Since ansibleapp assets got moved back to this repo, automated builds and pushes for relevant containers are broken. Needs to get fixed.

[11:35] < rhallisey> I can connect to dockerhub today
[11:35] < rhallisey> interesting
[11:36] < ernelson> I don't know what the deal is with that api, but it's very unreliable
[11:36] < rhallisey> we also should add some error checking around requests
[11:36] < rhallisey> yesterday I would get a 401 for each apb then 429s after docker got sick of the errors
[11:37] < rhallisey> 429 is too many requests and it would prevent me from trying again for a few minutes
[11:37] < ernelson> rhallisey: +1
[11:37] < ernelson> definitely need more validation
[11:37] < rhallisey> we should fail after 1 or 2 401's
[12:07] <@jmrodri> rhallisey: is this the broker that was making too many requests?
[12:07] < rhallisey> yes
[12:07] < rhallisey> broker would ask for image data from the dockerhub
[12:07] < ernelson> it's really easy to do, I'm rate limited all the time
[12:14] <@jmrodri> rhallisey: ok. definitely need to fix that then.
[12:14] <@jmrodri> ernelson: I'm sure

• We should require 2 people with merge access to ACK a PR. One reviewer can easily miss something, a second sanity check gives me a lot of confidence
• Above means we need to enumerate who those people are, either in the contributing doc or somewhere else. I noticed in origin they have "owners" files. A top level doc for us would probably suffice
• Should have a section encouraging new tests for new work, and explicitly state PRs that fail make test will not be reviewed until they are passing
• We should have a single script that runs all the preliminary source checks so that contributors can make sure their source is compliant and will pass that part of travis (without running regression stuff). Thinking vet, lint, format
• Could add a new makefile target, something like make check that runs tests and the source stuff. Add this to the contributing template to say: "Have you verified you pass make check?"
• Should talk about [DO NOT MERGE] [WIP] PRs. Please tag them as such to make sure they aren't reviewed and merged.

Got bit by this while debugging the new filter feature, I used "whitelist" instead of the expected "white_list". We should at least warn about unknown keys so users aren't expecting something like this to take when it's an unexpected key.

If a bind request is made against an instance, and no credentials have been returned from the apb during the provision or bind, we end up without a credentials object in etcd (as we should), and the broker actually segfaults somewhere around here:

https://github.com/fusor/ansible-service-broker/blob/master/pkg/broker/broker.go#L386

The segfault is absolutely a bug, because my intent as written was to handle this case, so it's not behaving as expected.

Is it okay for the catalog to request a bind against an instance that hasn't given the broker any credentials, either by choice or because of an error in the apb? How should the broker handle this?

With synchronous provisions, the provision call should return a dashboard_url. According to this PR it should also return dashboard_url during an async provision as well.

We shouldn't be doing this on every make build and it really throws a wrench into the dev workflow. Should vendor get triggered by a newer glide.yaml?

It would be nice to print the parameters from apbs to stdout. But, we can't do that unless we blockout passwords.

Current behavior:
#apb.yml
image: ansibleplaybookbundle/my-apb
This means that the broker downloads the spec from an organization, but may actually download the image from an entirely different organization, leading to confusion.

Proposed:
#apb.yml
image: my-apb
ASB will assume the apb image is in the same repository and org where the spec was found.

Will require changes in:

• ansible-service-broker - no longer use the org in a spec and use where it's hosted instead.
• ansible-playbook-bundle - remove org from apb-init and build

Reasoning:

1. Stable - Allow a user to download an image, retag it, and then push it to a different org for either testing or production use without modifying and rebuilding. We want to test the built image as is without modifying it and introducing changes.
2. Less confusion - since a user may forget to change the org, they'd be testing the original image on a different org and may be confused why changes aren't reflected.
3. Reusable - we would want users and developers to be sharing images without forcing them to rebuild the image with changes before pushing to their own organization for use.

Remove ProjectRoot since the only thing that uses it are tests.

$ git grep ProjectRoot
pkg/broker/util.go:func ProjectRoot() string {
pkg/broker/util_test.go:func TestProjectRoot(t *testing.T) {
pkg/broker/util_test.go:        ft.AssertEqual(t, ProjectRoot(), rootpath, "paths not equal")

When you run a provision, we output the number of retries left and the return of oc logs -f. Once the patch for the Kubernetes client merges, we can do a lot more. We can parse the pod status, see if the image is pulling, make a note what node the pod is on, ect...

[2017-05-22T17:07:33.705-04:00] [INFO] Container not up yet, retrying 1 of 150 on pod aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606
[2017-05-22T17:07:33.929-04:00] [DEBUG] oc log output:
Error from server: container "aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606" in pod "aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606" is waiting to start: ContainerCreating
[2017-05-22T17:07:33.929-04:00] [DEBUG] status: still waiting to start
[2017-05-22T17:07:39.929-04:00] [INFO] Container not up yet, retrying 2 of 150 on pod aa-ee5b096d-d46b-46ff-bcef-f2b89dab8606

All the clients: docker, etcd, kubernetes, and coming soon Openshift, are all created in different locations. In some cases, we're even creating the clients each time we perform an action. Let's pool all the client initialization into one directory so they can all be called at once.

@shawn-hurley welcomly (is that a word?) pointed out request.FormValue is a much better way of unwrapping request params. Let's update the handler layer to respect it.

Registries are produced via factory, but the RegistryConfig struct is currently trying to serve the needs of each, which means values that are required to be present in the registry section of the config are not used for some concrete registry adapters.

Need to get better with the types here, i.e.

  registry:
    type: dev
    fields:
       user: hello
       url: foo.bar.com

The runCommand function is how we a primarily creating pods and containers. So we need this function to be more accessible by other packages in the broker. One solution would be to create a utils pkg that contains this function.

func runCommand(cmd string, args ...string) ([]byte, error) {
	output, err := exec.Command(cmd, args...).CombinedOutput()
	return output, err
}

The APB name we generate is used to note the PodPreset/Binding/Secret being created by Service Catalog.

There is a problem running from ansibleplaybookbundle and doing a provision/bind of mediawiki and postgres.

When we do the binding and trigger the 2nd deployment of mediawiki it fails to come up.

The below error is present:

(combined from similar events): Error creating: Pod "mediawiki123-2-shk6c" is invalid: metadata.annotations: Invalid value: "podpreset.admission.kubernetes.io/dockerhub-ansibleplaybookbundle-rhscl-postgresql-apb-3xcwz-2v5r4": name part must be no more than 63 characters

Arose out of #222 discussion.

It sounds like most people are in favor of singleton clients.

Considerations:

• Needs testing WRT concurrent usage of a since client instance (for all relevant clients).
• Should the instances be initialized and validated on first use, or on app startup? I think @shawn-hurley made some good arguments for the latter, fail fast and early.

SSIA

Code Refactor

I think we're exposing to many functions between pkgs. For example, the broker class shouldn't need to do anything with the apb class except call bind, unbind, provision, deprovision. We shouldn't do anything in the dao class except post or gather etcd data. And the client class shouldn't do anything other than gather clients.

Proposed Work

• Let's define a set of functions that will be public per pkg
• Add a comment to the top of each pkg describing the available public functions and what they do
• Any additional functionality a pkg needs should be a private function added to that pkg or a new pkg should be created

Bind always returns 201 CREATED or 400 BADREQUEST. There are several other codes that need to be handled:

https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#binding

Responses with any other status code will be interpreted as a failure and an unbind request will be sent to the broker to prevent an orphan being created on the broker. Brokers can include a user-facing message in the description field; for details see Broker Errors.

Feature: Support other Clusters than OpenShift

The ansible-service-broker is currently OpenShift only because OpenShift has a few features that Kubernetes is still developing. The delta between OpenShift and Kubernetes is going to be a common occurrence so need a way to easily support both. I'm proposing that we can solve this by organizing our code paths into separate pieces and filling the gaps to meet the service-catalog specs:

• Kubernetes operations
• OpenShift operations
• Kubernetes hack operations that perform the actions done in 'OpenShift operations', but for Kubernetes

Proposed work

1. Split out all the clients into pkg/clients/
   1a. #213
   2a. #222
2. Split out all Cluster logic into a pkg/cluster/openshift.go and pkg/cluster/kubernetes.go

• Kubernetes is used for the majority of the broker actions while OpenShift a small number of actions. Let's break apart these code paths and call into a cluster pkg for cluster actions.

3. Create a pkg/cluster/kubernetes-hack.go that will do OpenShift specific functions the Kubernetes way
4. Use a Kubernetes solution to service accounts
   4a. https://github.com/openshift/ansible-service-broker/blob/master/pkg/apb/svc_acct.go
5. Change APBs so they support Kubernetes
   5a. fusor/apb-examples#60
6. Remove all "oc" commands and replace them with Kubernetes and OpenShift client calls
   6a. https://github.com/openshift/ansible-service-broker/search?p=1&q=%22oc%22&type=&utf8=%E2%9C%93

Right now there is too much variation between the local and incluster code paths. We want to use nearly identical code paths so that when we develop the broker locally, it will reflect how it will run incluster.

Provision could call DeepEqual twice which can be expensive.

https://github.com/fusor/ansible-service-broker/blob/master/pkg/broker/broker.go#L197-L217

Also remove the logging of parameters which could contain sensitive information.

Some background: today, we merged a major breaking change to the apb.yml schema (multi-plan support). It was a case where the apb.yml schema had changed, and the broker needed an overhaul to support it. This meant that brokers prior to the merge were not able to deploy APBs that were older than the apb-examples PR, and vice versa. This was a bit of a special case, because we had previously been faking out a core piece of the OSB spec (plans), but I suspect it will remain a problem going forwards. If we ever make breaking, required changes to apb.yml contents, we're going to end up in the same situation.

This kicked in CI, which rebuilt canary APBs, but latest was not rebuilt. This meant I needed to test against canary APBs, but right now, there is no easy way to configure a broker to deploy from a particular tag.

This is definitely related to #288. The APBs shouldn't care at all about where they live; it breaks portability.

Scope for this issue is, we should determine what belongs in an apb.yml schema, and what should be configurable in the broker so allow for ease of deploying APBs from alternative locations.

Arose out of #222 discussion.

Client struct in apb/client.go simply aggregates references to the various clients, which doesn't provide much value. It should be removed along with the NewClient constructor.

Additionally, pushing all the client related logic into the clients pkg brings into question whether apb/client.go is really an appropriate name for the file. Given the work being done (actually executing apb methods), consider a better name RunApb(method, params).

Steps to recreate:

1. Use catasb 'dev' branch
2. Create a new project: postgres-demo-apb
3. Launch postgres-demo-apb into the project postgres-demo-apb
4. Launch a new python app from source https://github.com/fusor/awsdemo.git into postgres-demo-apb
5. After they are provisioned
6. Click on python app and select Create Binding to the postgres-demo-apb

(Note: As of 6/7 we lack parameter schema support, so we are not reading in the namespace passed into us, hence a workaround for testing further is to manually create the namespace postgres-demo-apb
since the postgres-demo-apb has a default value of namespace of that.)

This line is
https://github.com/openshift/ansible-service-broker/blob/master/pkg/broker/broker.go#L319
params := make(apb.Parameters)
params["provision_params"] = *instance.Parameters

Below is the core dump from the 'asb' image:

172.17.0.1 - - [07/Jun/2017:13:38:12 +0000] "GET /v2/catalog HTTP/1.1" 200 14224
[2017-06-07T13:39:12.209Z] [DEBUG] Dao::GetRaw [ /service_instance/d2b660c6-739e-4b75-854e-4ebd1acd903d ] -> [ {"id":"d2b660c6-739e-4b75-854e-4ebd1acd903d","spec":{"id":"26355d1a-0301-4841-85af-536a4d3afaa9","name":"postgresql-demo-apb","image":"ansibleplaybookbundle/postgresql-demo-apb","tags":["database"],"bindable":true,"description":"PostgreSQL apb implementation","metadata":{"displayName":"Postgresql demo","documentationUrl":"","imageUrl":"https://upload.wikimedia.org/wikipedia/commons/thumb/2/29/Postgresql_elephant.svg/64px-Postgresql_elephant.svg.png","longDescription":"An apb demo that deploys postgresql and loads it with sample data"},"async":"optional","parameters":[{"name":"namespace","description":"Namespace to deploy the cluster to","type":"string","required":false,"default":"postgresql-demo-apb"},{"name":"postgresql_database","description":"postgresql database name","type":"string","required":false,"default":"admin"},{"name":"postgresql_password","description":"postgresql database password","type":"string","required":false,"default":"admin"},{"name":"postgresql_user","description":"postgresql database username","type":"string","required":false,"default":"admin"}]},"parameters":null} ]
2017/06/07 13:39:12 http: panic serving 172.17.0.1:54134: runtime error: invalid memory address or nil pointer dereference
goroutine 1601 [running]:
net/http.(*conn).serve.func1(0xc4200da000)
/usr/lib/golang/src/net/http/server.go:1491 +0x12a
panic(0x177f2c0, 0xc420014060)
/usr/lib/golang/src/runtime/panic.go:458 +0x243
github.com/openshift/ansible-service-broker/pkg/broker.AnsibleBroker.Bind(0xc42065d0e0, 0xc42019fdd0, 0xc420145cf0, 0xf, 0xc420145d20, 0x5, 0xc420145d48, 0x5, 0x2645ac0, 0xc420316060, ...)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/pkg/broker/broker.go:319 +0x228
github.com/openshift/ansible-service-broker/pkg/broker.(*AnsibleBroker).Bind(0xc420316180, 0xc42077c100, 0x10, 0x10, 0xc42077c110, 0x10, 0x10, 0xc4200da200, 0x1a2b1a1, 0x1a2b1a0, ...)
:9 +0xea
github.com/openshift/ansible-service-broker/pkg/handler.handler.bind(0x0, 0x0, 0x0, 0x0, 0xc420366c00, 0x5, 0x8, 0xc4203ab2c0, 0x0, 0x26571a0, ...)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/pkg/handler/handler.go:188 +0x2ca
github.com/openshift/ansible-service-broker/pkg/handler.(handler).(github.com/openshift/ansible-service-broker/pkg/handler.bind)-fm(0x7f2efe0249e0, 0xc420b18180, 0xc420e6c1e0, 0xc420b181b0)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/pkg/handler/handler.go:53 +0x80
github.com/openshift/ansible-service-broker/pkg/handler.createVarHandler.func1(0x7f2efe0249e0, 0xc420b18180, 0xc420e6c1e0)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/pkg/handler/handler.go:33 +0x65
net/http.HandlerFunc.ServeHTTP(0xc420436640, 0x7f2efe0249e0, 0xc420b18180, 0xc420e6c1e0)
/usr/lib/golang/src/net/http/server.go:1726 +0x44
github.com/openshift/ansible-service-broker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc420971a78, 0x7f2efe0249e0, 0xc420b18180, 0xc420e6c1e0)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/vendor/github.com/gorilla/mux/mux.go:114 +0x10d
github.com/openshift/ansible-service-broker/pkg/handler.handler.ServeHTTP(0x0, 0x0, 0x0, 0x0, 0xc420366c00, 0x8, 0x8, 0xc4203ab2c0, 0x0, 0x26571a0, ...)
/home/jesusr/dev/src/github.com/openshift/ansible-service-broker/pkg/handler/handler.go:69 +0x52
github.com/openshift/ansible-service-broker/pkg/handler.(*handler).ServeHTTP(0xc420316b40, 0x7f2efe0249e0, 0xc420b18180, 0xc420e6c000)
:2 +0x96

Let's create a configuration option so we can control this image pull policy for ABPs
https://github.com/openshift/ansible-service-broker/blob/master/pkg/apb/client.go#L172

IfNotPresent would help in several developer testing scenarios

The configurable pull policy is currently being set to 'Always', even though it's being configured to be set to "IfNotPresent".

Right now we have to different registry adapters for RHCC and Dockerhub. This is due to the fact that Dockerhub expects an authorization token workflow and RHCC has a different search mechanism. We found that under the hood both adapters are doing some introspection on the manifest for each image, they just both vary on how to find that manifest.

Moving forward we would like to consolidate the code so that a common registry package exists which does introspection on the manifest to return the spec and then individual adapters for sourcing the images to grab the manifests from.

We have 4 broker config files in the etc directory. IMO we should only carry one, but document each use case.

Most of our projects have the copyright at the top of each file. I'm proposing two options for the headers. The text is the same, only the format changes. The first one looks more readable IMO.

Option 1 - leading asterisks (*)

/*
 * Copyright (c) 2017 Red Hat, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 *
 * Red Hat trademarks are not licensed under Apache License, Version 2.
 * No permission is granted to use or replicate Red Hat trademarks that
 * are incorporated in this software or its documentation.
 */

Option 2 - no leader

/*
Copyright (c) 2017 Red Hat, Inc.
 
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at 
 
   http://www.apache.org/licenses/LICENSE-2.0
 
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
 
Red Hat trademarks are not licensed under Apache License, Version 2. 
No permission is granted to use or replicate Red Hat trademarks that
are incorporated in this software or its documentation.
*/

Option 3 - // leader

//
// Copyright (c) 2017 Red Hat, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//    http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// Red Hat trademarks are not licensed under Apache License, Version 2.
// No permission is granted to use or replicate Red Hat trademarks that
// are incorporated in this software or its documentation.
//

As part of make build we remove docker vendor dependency instead of having glide flatten it rm -rf ${GOPATH}/src/github.com/fusor/ansible-service-broker/vendor/github.com/docker/docker/vendor. We should get glide.yaml & glide.lock into good shape so we don't have to do this anymore and it avoid this coming up in the future.

Would be nice to have a spec compliance toolkit that we could point at a broker and verify the different return codes etc.

From discussion of #222

It's a vestige from when we pulled images before running then with oc run. I don't remember why we needed the pull in the first place, it's possible we never did. It's a use case entirely covered by the Pod pullPolicy. Huge bonus points if it means we can shed a dependency on docker entirely. We should definitely look into this, probably appropriate for an issue and follow up PR.

https://github.com/openshift/ansible-service-broker/blob/master/scripts/run_latest_build.sh

$ oc get pods
NAME READY STATUS RESTARTS AGE
asb-2357364550-krm3l 0/1 CrashLoopBackOff 2 49s
etcd-2338997634-jk8jv 0/1 CrashLoopBackOff 2 49s

$ oc logs asb-2357364550-krm3l
Using config file mounted to /etc/ansible-service-broker/config.yaml

== Starting Ansible Service Broker... ==

[2017-07-24T20:03:47.136Z] [NOTICE] Initializing clients...
[2017-07-24T20:03:47.136Z] [DEBUG] Trying to connect to etcd
[2017-07-24T20:03:47.136Z] [INFO] == ETCD CX ==
[2017-07-24T20:03:47.136Z] [INFO] EtcdHost: etcd
[2017-07-24T20:03:47.136Z] [INFO] EtcdPort: 2379
[2017-07-24T20:03:47.136Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-24T20:03:48.137Z] [ERROR] client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://etcd:2379 exceeded header timeout

$ oc logs etcd-2338997634-jk8jv
2017-07-24 20:03:11.733160 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PORT=2379
2017-07-24 20:03:11.733427 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT=2379
2017-07-24 20:03:11.733434 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_ADDR=172.30.147.171
2017-07-24 20:03:11.733451 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_HOST=172.30.147.171
2017-07-24 20:03:11.733458 W | pkg/flags: unrecognized environment variable ETCD_PORT=tcp://172.30.147.171:2379
2017-07-24 20:03:11.733462 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP=tcp://172.30.147.171:2379
2017-07-24 20:03:11.733467 W | pkg/flags: unrecognized environment variable ETCD_PORT_2379_TCP_PROTO=tcp
2017-07-24 20:03:11.733472 W | pkg/flags: unrecognized environment variable ETCD_SERVICE_PORT_ETCD_ADVERTISE=2379
2017-07-24 20:03:11.733493 I | etcdmain: etcd Version: 3.2.4
2017-07-24 20:03:11.733506 I | etcdmain: Git SHA: c31bec0
2017-07-24 20:03:11.733511 I | etcdmain: Go Version: go1.8.3
2017-07-24 20:03:11.733515 I | etcdmain: Go OS/Arch: linux/amd64
2017-07-24 20:03:11.733520 I | etcdmain: setting maximum number of CPUs to 16, total number of available CPUs is 16
2017-07-24 20:03:11.733959 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-07-24 20:03:11.735000 I | embed: listening for peers on http://localhost:2380
2017-07-24 20:03:11.735072 I | embed: listening for client requests on 0.0.0.0:2379
2017-07-24 20:03:11.736553 C | etcdserver: create snapshot directory error: mkdir /data/member/snap: permission denied

$ oc get pv | grep Bound
pv0073 100Gi RWO,ROX,RWX Recycle Bound ansible-service-broker/etcd 3m

$ oc describe pv pv0073
Name: pv0073
Labels: volume=pv0073
Annotations: pv.kubernetes.io/bound-by-controller=yes
StorageClass:
Status: Bound
Claim: ansible-service-broker/etcd
Reclaim Policy: Recycle
Access Modes: RWO,ROX,RWX
Capacity: 100Gi
Message:
Source:
Type: HostPath (bare host directory volume)
Path: /var/lib/origin/openshift.local.pv/pv0073
Events:

$ ls -larth /var/lib/origin/openshift.local.pv/pv0073
ls: cannot open directory '/var/lib/origin/openshift.local.pv/pv0073': Permission denied
jmatthews@beast r (master) $ sudo ls -larth /var/lib/origin/openshift.local.pv/pv0073
total 12K
drwxr-xr-x. 103 root root 4.0K Mar 15 12:29 ..
drwx------. 4 1000170000 root 4.0K Jul 21 10:01 member
drwxrwx---. 3 root root 4.0K Jul 24 16:05 .

APBs can run for a very short time ~30 seconds to a very long time ~30 minutes. The way we gather bind credentials has a timeout of 5 minutes. We need to allow for APBs to have all the time they needs to run.

Possible solutions:

1. Poll the APB and wait for an error or completion

• Remove the broker timeout entirely and expect that if an APB hasn't exited with an error or hasn't completed bind-init then work is still being done.
• An issue with this approach is that if a task hangs in a playbook, we have no timeout for it and the broker will forever look for the bind credentials.

2. Interrupt driven I/O

• Use entrypoint.sh to signal the broker that the playbook has completed.
• This could be a simple http request to the broker letting it know that now is a good time to gather credentials. In this scenario, there is no timeout on the APB and the Broker doesn't waste resources looking for credentials.