Running on bosh-lite errors when checking space guid about cf-smoke-tests

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/143158861

The labels on this github issue will be updated when the story is started.

[dsabeti]

Hey @bonzofenix. Could you show the full failure output? My suspicion is that some other part of the test is failing and preventing the test from targeting an org.

[bonzofenix]

Hi dsabeti,
Here is the complete output of the test: https://gist.github.com/bonzofenix/476fa832cae21d2a2fe2705b4fe5922e
Thank you

[dsabeti]

The output suggests that the test timed out setting the quota on the organization. The very next command that would have run (if cf set-quota had succeeded) is cf target. It seems pretty clear that the failure to set the quota is the real failure, and the failure to fetch the space guid is a side-effect.

[bonzofenix]

cf_dial_timeout_in_seconds is set 300. Seems like a good amount of time. We looked through the code and it seems that the timeout is hardcoded: https://github.com/cloudfoundry/cf-smoke-tests/search?utf8=%E2%9C%93&q=DEFAULT_TIMEOUT&type=

[dsabeti]

Yeah, I don't know that opening the timeout will help. Have you tried running cf set-quota by hand? How long does it take? If it sometimes exceeds 30 seconds, you'll need to figure out why it takes so long. Some things to consider:

• Is there something going wrong with your network?
• Is the Cloud Controller experiencing significant load or facing resource limits?
• Does the CC database respond to SQL commands in a reasonable amount of time?

[bonzofenix]

Ok you are right I see whats happening, seems that the 30 seconds timeout is not enough:

Can we make this constant configurable?
Do you know how I could make my CF bosh-lite deployment run faster so that the smoke test would go green consistently?

[dsabeti]

@bonzofenix We can make that timeout configurable, but 30 seconds is pretty bad for a simple API call. In my comment above, I specified a few things to take a look at (CC load, CCDB slowness).

You might also compare it to other CC endpoints. For example, does cf curl /v2/info take a long time? If so, there's something pretty bad going on with either your network or the CC.

For networking slowness, I'd suggest debugging with traceroute or tcpdump.

[dsabeti]

Hey @bonzofenix, any update on this?

[dsabeti]

I'm going to close the issue due to inactivity, but feel free to re-open it if you have more questions.