rabbitmq-server job pre-start script fail during deployment about cf-rabbitmq-release

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[blgm]

@wenxin-liu, which stemcell are you using?

The error message seems to come from deep within Erlang: https://github.com/erlang/otp/blob/master/lib/kernel/src/inet_config.erl#L260
@hairyhum @michaelklishin, would you expect the ~ts~n^M to be interpolated into something more meaningful? It looks like an attempt to do a printf equivalent.

My guess would be that this is caused by parsing the file /var/vcap/store/rabbitmq/erl_inetrc. A good one looks like:

{host, {10,0,4,10}, ["eaee322bf5da343c18b95dcb08dabad5"]}.
{host, {10,0,4,11}, ["dce07ff98e3da0dcee2c738aabbb8f1e"]}.
{host, {10,0,4,12}, ["881881c9ed15cef3c0417de34b239e29"]}.
{lookup, [file, native]}.

What does your file look like?

[wenxin-liu]

Hey @blgm and everyone,

Thanks for your suggestion, we've just recently managed to work around the problem, and indeed as you pointed out, it was to do with a syntax error in the erl_inetrc file. We're currently using ubuntu-trusty/v3586.36.

Although we've managed to work around the issue, although the core problem is not fixed. The core problem is: The release does not work when using a dynamic bosh network.

We are hosted in GCP and therefore use a dynamic network (defers IP selection to the IaaS provider). During deployment of the rmq release, it fails because the erl_inetrc file created needs to have the hosts information in the format of {host, IP, alias}. But when using a dynamic network, during deployment bosh only 'knows' the DNS names of the vms in the cluster, not the IPs. As a result, the resulting erl_inetrc file is in the format of {host, DNS name, alias}. The pre-start script then fails, and we get an error that the file cannot be parsed because it has a syntax error and it's not sorted in the right order. We discovered that if we switch to using a manual network, because bosh assigns the IPs, it also 'knows' the IPs and is able to put them into the erl_inetrc file as required. The resulting erl_inetrc file has the correct format, that is {host, IP, alias}. This works around the issue. But ideally this release should either fetch the IPs in a different way (currently the release uses Bosh links), or even better, not use IPs at all but DNS addresses (as using IPs means tightly coupled).

[hairyhum]

@blgm I'm not sure how the log message ends up in the stderr, but it seems like it's formatted correctly to the stdout.

[nodo]

Thanks for raising this and for sharing your investigation @wenxin-liu !

We have added a note to the README, mentioning that the release has only been tested using "manual" networks. We are planning to add support for BOSH DNS soon, hopefully after that we will be able to support other network types as well.

We are going to close the issue for now. Feel free to re-open if you have further questions.

@ferozjilla and @nodo

[wenxin-liu]

Thanks @nodo !