cloud_controller_api_health_check is flakey about capi-release

[selzoc]

I believe we have fixed this issue in d18bed8

[ishustava]

@selzoc yep, it was most prominent on bosh-lite.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/157404037

The labels on this github issue will be updated when the story is started.

[tcdowney]

Additional context

For future CAPI team members, this is how the Cloud Controller hooks up its health check script with the Route Registrar:

1. Cloud Controller uses a health check script that is generated by this template
2. The CF route-registrar uses this code to run the health check script above

Reproduction Steps

I was able to trigger this somewhat on demand by manually patching /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/runtime/info_controller.rb to sleep for 21 second (slightly longer than cf-deployment's timeout) 50% of the time.

diff --git a/app/controllers/runtime/info_controller.rb b/app/controllers/runtime/info_controller.rb
index 9da7821ac..7a489f570 100644
--- a/app/controllers/runtime/info_controller.rb
+++ b/app/controllers/runtime/info_controller.rb
@@ -4,6 +4,8 @@ module VCAP::CloudController

     get '/v2/info', :read
     def read
+      chance =  [1, 2].sample
+      sleep(21)  unless chance == 2
       info = {
         name: @config.get(:info, :name),
         build: @config.get(:info, :build),

Then while running tail -f /var/vcap/sys/log/route_registrar/route_registrar.log I see the Cloud Controller's route waffle between being registered and unregistered.

Adding some retries and explicit timeouts to our health check script makes this less likely (I didn't observe it get unregistered during my manual experimentation). This might look something like this:

curl --retry 3 --retry-delay 1 --max-time 5 http://localhost:9022/v2/info

Our monit health check that functions similarly, will wait up to sixty seconds for 5 monit cycles:

capi-release/jobs/cloud_controller_ng/monit

Lines 36 to 37 in c729f53

	and request '/v2/info'
	with timeout 60 seconds for 5 cycles

[tcdowney]

@ishustava @njbennett

Occasionally the CAPI healthcheck fails, for reasons that seem to have nothing to do with the status of the API itself.

We'd like to understand a little better how and why the curl is failing in your environment since a healthy API shouldn't take over. Is your environment under a significant amount of load? Do other requests get fielded in a reasonable amount of time? My repro steps above are a bit contrived and it's possible that adding retries might just exacerbate the actual underlying issue.

[selzoc]

Bumping this for question above ^^
@ishustava @njbennett

[ishustava]

Hey @tcdowney @selzoc,

Sorry for the delay.

We don't have any more error output other than the route_registrar.log posted above. At the time we could not find errors in cloud controller logs, so I suspect that curl might have failed for other reasons.

Unfortunately, we could not get enough information from the logs to be able to reproduce the problem and were hoping that we can add retries as a preventative measure.

[selzoc]

@ishustava @njbennett was this on a bosh lite? Could make sense that what @tcdowney was asking about a healthy api node taking over wouldn't apply in that case since the instance group is scaled to 1.