blobstore job fails to start after vm crash or reboot about capi-release

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/126686223

The labels on this github issue will be updated when the story is started.

[sykesm]

It appears the same problem exists with the cloud controller:

[2016-07-20 14:21:03+0000] ------------ STARTING cloud_controller_worker_ctl at Wed Jul 20 14:21:03 UTC 2016 --------------
[2016-07-20 14:21:03+0000] /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl: line 40: /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_2.pid: No such file or directory
[2016-07-20 14:22:43+0000] ------------ STARTING cloud_controller_worker_ctl at Wed Jul 20 14:22:43 UTC 2016 --------------
[2016-07-20 14:22:43+0000] /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl: line 40: /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_1.pid: No such file or directory
[2016-07-20 14:23:13+0000] ------------ STARTING cloud_controller_worker_ctl at Wed Jul 20 14:23:13 UTC 2016 --------------
[2016-07-20 14:23:13+0000] /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl: line 40: /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_2.pid: No such file or directory
[2016-07-20 14:24:53+0000] ------------ STARTING cloud_controller_worker_ctl at Wed Jul 20 14:24:53 UTC 2016 --------------
[2016-07-20 14:24:53+0000] /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl: line 40: /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_1.pid: No such file or directory
[2016-07-20 14:25:23+0000] ------------ STARTING cloud_controller_worker_ctl at Wed Jul 20 14:25:23 UTC 2016 --------------
[2016-07-20 14:25:23+0000] /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl: line 40: /var/vcap/sys/run/cloud_controller_ng/cloud_controller_worker_2.pid: No such file or directory

[sykesm]

Similar issues in consul: cloudfoundry-attic/consul-release#31

[sax]

We've made fixes in these two commits:
cloudfoundry/cloud_controller_ng@232f748
2cc5a9b

This will ensure that the directories in /var/vcap/sys/run are recreated when services start after a reboot.

When we switched our start commands to run as non-root, we moved all directory creation into pre-start, because we ran into problems with /var/vcap/sys/log not giving write permission to non-root users. It turns out that /var/vcap/sys/run gives write permission to the vcap group, so we can make those directories in start scripts.

Let us know if this fixes the issue for you, and close the issue if it does!

@sax && @adowns01

[sykesm]

Thanks.