rabbitmq / rabbitmq-clusterer

This project is ABANDONWARE. Use https://www.rabbitmq.com/cluster-formation.html instead.

RabbitMQ Clusterer

This plugin is no longer maintained, and completely unnecessary with supported releases of RabbitMQ.

This plugin provided an alternative means for creating and maintaining RabbitMQ clusters. It is highly opinionated and was created with specific opinionated infrastructure provisioning tooling in mind. Team RabbitMQ considers it to be a failed experiment and highly recommends against it.

Please upgrade to a supported RabbitMQ version instead. It provides a peer discovery subsystem introduced in RabbitMQ 3.7.0 or its predecessor, rabbitmq-autocluster. That plugin is not a strict alternative to this one but targets a wider range of provisioning scenarios.

Project status

The plugin was created to handle arbitrary order on nodes restart. Since RabbitMQ version 3.6.7 this problem is addressed in the core.

This plugin is considered deprecated, and it's recommended to switch to RabbitMQ's built-in cluster formation feature in order to avoid known issues that this plugin's opinionated behavior entails (such as #7).

Overview

Traditional RabbitMQ clustering is not very friendly to infrastructure automation tools such as Chef, Puppet or BOSH. The existing tooling (rabbitmqctl join_cluster and friends) is imperative, requires more oversight and does not handle potentially random node boot order very well. The Clusterer has been specifically designed with automated deployment tools in mind.

Unlike the existing tooling, the Clusterer is declarative and goal directed: you tell it the overall shape of the cluster you want to construct and the clusterer tries to achieve that. With the Clusterer, cluster configuration can be provided in a single location (a configuration file).

With rabbitmq-clusterer, nodes in a cluster can be restarted in any order, which can be the case with automation tools performing upgrades/reconfiguration, or due to node failure timing.

Project Maturity

This plugin is ancient abandonware. Do not use. Yes, really, it no longer has any reasons to exist.

Compatibility With Traditional RabbitMQ Clustering

The Clusterer is not generally compatible with the existing clustering tool-set. Do not use any of the rabbitmqctl commands relating to changing clusters: join_cluster, change_cluster_node_type, and update_cluster_nodes must not be used. If you do use these, this plugin likely won't be able to perform its jobs.

rabbitmqctl cluster_status may be used to inspect a cluster state, but the Clusterer sends to the standard Rabbit log files details about any clusters it joins or leaves. See the Inspecting the Clusterer Status section further down.

rabbitmqctl stop_app, rabbitmqctl forget_cluster_node, and rabbitmqctl start_app can be used to force a node out of a cluster before cluster config can be changed. While this is not generally recommended, there can be valid reasons for doing so, e.g. node running out of disk space and/or needing replacement for other reasons.

cluster_nodes in the RabbitMQ config file is incompatible with this plugin and must not be used.

Installation

Binary builds of this plugin are available from

• Bintray (like with other RabbitMQ Community Plugins)
• GitHub releases page

As with all other plugins, you must put the plugin archive (.ez) in the RabbitMQ plugins directory and enable it with rabbitmq-plugins enable rabbitmq_clusterer --offline.

For Recent RabbitMQ Versions (3.5.4 and Later)

Compiled plugin file needs to be placed into .

To use the plugin, it is necessary to override RABBITMQ_BOOT_MODULE to rabbit_clusterer. This is done similarly to other RabbitMQ environment variables.

Because this plugin coordinates RabbitMQ node start, it needs to be manually added to the Erlang VM code path:

export RABBITMQ_BOOT_MODULE=rabbit_clusterer
export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa /path/to/rabbitmq/plugins/rabbitmq_clusterer.ez/rabbitmq_clusterer-{clusterer-version}/ebin"

where {clusterer-version} is the build of the plugin (see GitHub releases page and Bintray):

export RABBITMQ_BOOT_MODULE=rabbit_clusterer
export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa /path/to/rabbitmq/plugins/rabbitmq_clusterer-1.0.3.ez/rabbitmq_clusterer-1.0.3/ebin"

Since .ez files are .zip archives, they can be easily inspected when you are not sure about the exact name of the directory you the file you've downloaded.

For RabbitMQ 3.4.x

rabbitmq-server Patch

With RabbitMQ versions earlier than 3.5.4, it is necessary to apply a rabbitmq-server patch and re-compile the broker.

The patch is provided in rabbitmq-clusterer/rabbitmq-server.patch. Change into the server scripts directory and apply it with:

patch -p1 < rabbitmq-clusterer/rabbitmq-server.patch

The patch assumes the plugin archive is at ${RABBITMQ_PLUGINS_DIR}/rabbitmq_clusterer.ez.

Usage in Environments with Dynamic Hostnames (e.g. Kubernetes)

Since this plugin assumes that all cluster members are known ahead of time and listed in the config, environments with dynamically generated hostnames must be configured to use known (or completely predictable) hostnames.

For Kubernetes specifically, there's an example repository contributed by Matt Friendman.

There's also another example that uses Kubernetes 1.5.x.

Cluster Config Specification

The Clusterer will communicate a new valid config to both all the nodes of its current config, and in addition to all the nodes in the new config. Even if the cluster is able to be formed in the absence of some nodes indicated in the config, the nodes of the cluster will continue to attempt to make contact with any missing nodes and will pass the config to them if and when they eventually appear.

All of which means that you generally only need to supply new configs to a single node of any cluster. There is no harm in doing more than this. The Clusterer stores on disk the currently applied config (it stores this next to the mnesia directory Rabbit uses for all its persistent data) and so if a node goes down, it will have a record of the config in operation when it was last up. When it comes back up, it will attempt to rejoin that cluster, regardless of whether this node was ever explicitly given this config.

There are a couple of ways to specify a cluster config: via an external file or inline.

Using External Config File

In a rabbitmq_clusterer section in rabbitmq.config file you can add a config entry that is a path to configuration file.

Below are some examples.

When using rabbitmq.conf (currently only available in RabbitMQ master):

clusterer.config = /path/to/my/cluster.config

When using the classic configuration format (rabbitmq.config, prior to 3.7.0) or advanced.config:

  [{rabbitmq_clusterer,
      [{config, "/path/to/my/cluster.config"}]
   }].

Like with rabbitmq.config or any other Erlang terms file, the dot at the end is mandatory.

Using Inline Configuration in rabbitmq.config

It is possible to provide cluster configuration in rabbitmq.config.

In rabbitmq.conf:

clusterer.version = 43
clusterer.nodes.disc.1 = rabbit@hostA
clusterer.nodes.disc.2 = rabbit@hostD
clusterer.nodes.ram.1  = rabbit@hostB
clusterer.gospel.node = rabbit@hostD

Or, using the classic config format (rabbitmq.config, prior to 3.7.0) or advanced.config:

  [{rabbitmq_clusterer,
      [{config,
          [{version, 43},
           {nodes, [{rabbit@hostA, disc}, {rabbit@hostB, ram}, {rabbit@hostD, disc}]},
           {gospel, {node, rabbit@hostD}}]
       }]
   }].

This approach makes configuration management with tools such as Chef somewhat less convenient, so external configuration file is the recommended option.

Using rabbitmqctl eval

rabbitmqctl eval 'rabbit_clusterer:apply_config().'

This will only have any effect if there is an entry in the rabbitmq.config file for the Clusterer as above, and a path is specified as the value rather than a config directly.

If that is the case, then this will cause the node to reload the file containing cluster config and apply it. Note that you cannot change the path itself in the rabbitmq.config file dynamically: neither Rabbit nor the Clusterer will pick up any changes to that file without restarting the whole Erlang node.

rabbitmqctl eval 'rabbit_clusterer:apply_config("/path/to/my/other/cluster.config").'

This will cause the Clusterer to attempt to load the indicated file as a cluster config and apply it. Using this method rather than the above allows the path to change dynamically and does not depend on any entries in the rabbitmq.config file. The path provided here is not retained in any way: providing the path here does not influence future calls to rabbit_clusterer:apply_config(). - using rabbit_clusterer:apply_config(). always attempts to inspect the path as found in rabbitmq.config when the node was started.

Note if you really want to, rather than suppling a path to a file, you can supply the cluster config as a proplist directly, just as you can in the rabbitmq.config file itself.

Cluster Configuration

A cluster config is an Erlang proplist consisting of just four tuples. The config can be supplied to Rabbit in a variety of ways and it is in general only necessary to supply a config to a single node of a cluster: the Clusterer will take care of distributing the config to all the other nodes as necessary.

[{version, 43},
 {nodes, [{rabbit@hostA, disc}, {rabbit@hostB, ram}, {rabbit@hostD, disc}]},
 {gospel, {node, rabbit@hostD}}].

The above gives an example cluster config. This specifies that the cluster is formed out of the nodes rabbit@hostA, rabbit@hostB and rabbit@hostD and that rabbit@hostA and rabbit@hostD are disc nodes and rabbit@hostB is a ram node. The nodes tuple is really the only tuple that describes the shape of the cluster. The other tuples describe how to achieve the cluster, and are thus mainly irrelevant once the cluster has been achieved.

In general, the Clusterer will wait indefinitely for the conditions to be correct to form any given cluster. This is in contrast to the existing tools which will either timeout or in some cases take (arguably) unsafe actions. For example, the existing tools will allow a fresh node to fully start when it is supplied with a cluster configuration which involves other nodes which are not currently contactable. This is unsafe because those other nodes might not be fresh nodes: the intention would be for the fresh node to sync with those other nodes and preserve the data those nodes hold. When those other nodes eventually return, manual intervention is then required to throw away some data and preserve others. The Clusterer, by contrast, would wait until it could either verify that all the nodes to be part of the cluster are fresh (so there is no data to preserve at all), or failing that would wait until one of the non-fresh nodes was fully up and running, at which point it could sync with that node.

• version: non negative integer

  All configs are versioned and this is used to decide which of any two configs is the youngest. A config which has a smaller version number is older. Configs will be ignored unless they are younger than the current config. Note that in lieu of any config being provided by the user, the default config is used which has a version of 0. Thus user supplied configs should use a version of 1 or greater.

• nodes: list

  List the names of the nodes that are to be in the cluster. If you list node names directly then they are considered to be disc nodes. If you specify nodes by using a tuple, you can specify a disc node using either disc or disk. If you want to specify ram nodes, you must use a tuple, with ram as the second element. Order of nodes does not matter. The following are all equivalent.

    {nodes, [rabbit@hostA, rabbit@hostD, {rabbit@hostB, ram}]}
    {nodes, [rabbit@hostD, rabbit@hostA, {rabbit@hostB, ram}]}
    {nodes, [{rabbit@hostB, ram}, rabbit@hostD, rabbit@hostA]}
    {nodes, [rabbit@hostA, {rabbit@hostD, disk}, {rabbit@hostB, ram}]}
    {nodes, [{rabbit@hostA, disc}, {rabbit@hostD, disk}, {rabbit@hostB, ram}]}
  

• gospel: reset or {node, nodename}

  When multiple nodes are to become a cluster (or indeed multiple clusters are to merge: you can think of an unclustered node as a cluster of a single node) some data must be lost and some data can be preserved: given two unclustered nodes A and B that are to become a cluster, either A's data can survive or Bs data can survive, or neither, but not both. The gospel` tuple allows you to specify which data should survive:

  • reset will reset all nodes in the cluster. This will apply every time the cluster config is changed and applied (i.e. if you change some other setting in the config, bump the version number, leave the gospel as reset and apply the config to any node in your cluster, you will find the entire cluster resets). This is deliberate: it allows you to very easily and quickly reset an entire cluster, but in general you'll only occasionally want to set gospel to reset.

  • {node, nodename} The nodename must appear in the nodes tuple. The data held by the existing cluster of which nodename is a member will survive. Nodes that are listed in the nodes tuple but which are not currently members of the same cluster as nodename will be reset. The phrasing here is very deliberate: it is not necessary for nodename to actually be up and running for this to work. If you have an existing cluster of nodes A and B and you want to add in node C you can set the gospel to be {node, A}, add C to the nodes tuple, bump the version and apply the config to C and provided at least one of A or B is up and running, C will successfully cluster. I.e. if only B is up, B still knows that it is clustered with A, it just happens to be the case that A is currently unavailable. Thus C can cluster with B and both will happily work, awaiting the return of A.

    In this particular case, the subsequent behaviour when A returns is important. If A has been reset and is now running an older config then it is A that is reset again to join back in with B and C. I.e. the gospel setting is really identifying that the data that A holds at a particular moment in time is the data to be preserved. When A comes back, having been reset, A realises that the gospel is indicating an older version of A, which is preserved by the surviving cluster nodes of B and C, not the newer reset data held by A. The upshot of this is that in your cluster, if a node fails, goes down and has to be reset, then to join it back into the cluster you don't need to alter anything in the cluster config (and indeed shouldn't): even if the failed node was named as the gospel, you shouldn't make any changes to the config.

    By contrast, if A comes back and has been reset but is now running a younger config than B and C, then that younger config will propagate to B and C. If A is named as the gospel in the new younger config, then that refers to the data held by the new younger A, and so B and C will reset as necessary.

Mistakes in config files

Config files can contain mistakes. If you apply a config file using rabbitmqctl eval then you'll get feedback directly. If you specify the config file via rabbitmq.config then and mistakes will be logged to Rabbit's log files.

In general, the Clusterer tries reasonably hard to give informative messages about what it doesn't like, but that can only occur if the config is syntactically valid in the first place. If you forget to bump the version number it will complain, and generally whenever the Clusterer comes across configs with equal version numbers but semantically different contents it takes highly evasive action: in some situations, it may decide to shut down the whole Erlang node immediately. It is your responsibility to manage the version numbers: the Clusterer expects to be able to order configs by version numbers, and thus determine the youngest config. You need to ensure it can do this. If you're building cluster configs automatically, one sensible approach would be to set the version to the number of seconds since epoch, for example.

Inspecting the Clusterer Status

rabbitmqctl cluster_status presents basic information about clusters, but does not interact with the Clusterer. rabbitmqctl eval 'rabbit_clusterer:status().', on the other hand, does, and shows which config is in operation by the node and what the Clusterer is trying to do. If the cluster has been established then the command will also display which nodes are known to be currently up and running.

Building From Source

The Clusterer reuses parts of the RabbitMQ umbrella repository. Before building the plugin, make sure it is cloned as rabbitmq_clusterer under it, much like other plugins.

To build the plugin run make. The VERSION environment variable is used to specify plugin version, e.g.:

VERSION=3.6.6 make

To package the plugin run make dist. In some cases, make clean dist is the safest option.

Linking in Development Environment

If you're running a development environment and want to link through from the rabbit/plugins directory, link to rabbitmq_clusterer/plugins/rabbitmq_clusterer-$VERSION.ez. Do not just link to the rabbitmq_clusterer directory.

License and Copyright

(c) 2013-2017 Pivotal Software Inc.

rabbitmq-clusterer's Issues

rabbitmq-server patch question

Hi.
When I want to use this plugin with rabbitmq-server 3.5.3 what is the easiest way to patch server?
Should i build server from sources, or I can install it from packet? If I decide to build and install server from sources, what would be the correct way to do it? If I install it from package how should I use your patch for rabbitmq-server?

Initial configuration of RabbitMQ nodes

Hi! I have a question on the initial configuration which should be passed to the clusterer plugin.

Suppose I create a number of RabbitMQ nodes, lets say N, all at the same time. I do not know the node names of individual nodes in advance so I can't populate the config upfront.

What should be the initial config for individual nodes, so that they would wait for the actual config to be provided?

I could experiment to find out about that but I'd like to know what's the official way of doing this.

BTW, thanks for this plug-in. It's certainly something which should become the new official clustering method some time soon. Keep up the good work.

Fails with 'init terminating in do_boot ()' in RabbitMQ 3.6.9

I've searched high and low for a way to debug this but I cannot seem to get any information on what would be causing this. I'm attempting to cluster 3 RabbitMQ 3.6.9 nodes in Kubernetes using rabbitmq-clusterer but I am running into an issue where rabbitmq-server fails and prints a pretty unhelpful error message.

Setup:
I am using the rabbitmq-clusterer .ez file from here: https://www.rabbitmq.com/community-plugins/v3.6.x/rabbitmq_clusterer-3.6.x-667f92b0.ez (I have used the release on BinTray as well, same issue)
RabbitMQ Server v 3.6.9

Steps to reproduce:
Here is a simple dockerfile which highlights the issue:

FROM rabbitmq:3.6.9-management

ENV RABBITMQ_BOOT_MODULE=rabbitmq_clusterer \
    RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa /plugins/rabbitmq_clusterer-3.6.9.ez/rabbitmq_clusterer-3.6.x-667f92b0/ebin"

ADD rabbitmq_clusterer-3.6.9.ez /plugins/rabbitmq_clusterer-3.6.9.ez

RUN apt-get update \
 && apt-get install -y jq curl unzip nano \
 && rabbitmq-plugins enable --offline rabbitmq_clusterer

When run this produces an Erlang crash with the message:

{"init terminating in do_boot",{undef,[{rabbitmq_clusterer,boot,[],[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ()

Crash dump is being written to: erl_crash.dump...done

When I modify rabbitmq-server to print commands as they are executed here is the offending erl invocation:

RABBITMQ_CONFIG_FILE=/etc/rabbitmq/rabbitmq ERL_MAX_ETS_TABLES=50000 exec erl -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.9/ebin -noinput -s rabbitmq_clusterer boot -sname rabbit@5bc2e8bf26f9 -boot start_sasl -config /etc/rabbitmq/rabbitmq +W w +A 64 +P 1048576 +t 5000000 +stbt db +zdbbl 32000 +K true -kernel inet_default_connect_options [{nodelay,true}] -pa /plugins/rabbitmq_clusterer-3.6.9.ez/rabbitmq_clusterer-3.6.x-667f92b0/ebin -sasl errlog_type error -sasl sasl_error_logger tty -rabbit error_logger tty -rabbit sasl_error_logger tty -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/plugins:/usr/lib/rabbitmq/lib/rabbitmq_server-3.6.9/plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@5bc2e8bf26f9-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@5bc2e8bf26f9" +B i -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

Nothing is printed to RabbitMQ logs

Where this config goes wrong?

Thanks for this great job, but I encountered some problems in using it properly, pls help.

I replaced the rabbitmq.config with the following content:

[
  { rabbit, [ { loopback_users, [ ] } ] },
  { rabbitmq_clusterer, [ { config, "/etc/rabbitmq/rabbitmq-clusterer.config" } ] }
].

Then the rabbitmq-clusterer.config with the following content:

[{rabbitmq_clusterer,
    [{config,
        [
            {version: 1},
            {nodes, [
                {rabbit@rabbitmq-server-0, disc},
                {rabbit@rabbitmq-server-1, ram},
                {rabbit@rabbitmq-server-2, disc}
            ]},
            {gospel, {node, rabbit@rabbitmq-server-0}}
        ]
    }]
}].

Then exported environment variables:

RABBITMQ_BOOT_MODULE as    rabbit_clusterer
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS as    -pa /plugins/rabbitmq_clusterer-1.0.2.ez/3.5.x-189b3a81/ebin

Then enabled the plugins and tried to start, but get he following errors:

rabbitmq-server-2    | {"init terminating in do_boot",{{badmatch,{error,{not_started,rabbit_common}}},[{rabbit_clusterer,boot,0,[{file,"src/rabbit_clusterer.erl"},{line,31}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
rabbitmq-server-2    | init terminating in do_boot ()
rabbitmq-server-2    |
rabbitmq-server-2    | Crash dump is being written to: erl_crash.dump...rabbitmq-server-2 exited with code 1
rabbitmq-server-1    | init terminating in do_boot ()
rabbitmq-server-1    | {"init terminating in do_boot",{{badmatch,{error,{not_started,rabbit_common}}},[{rabbit_clusterer,boot,0,[{file,"src/rabbit_clusterer.erl"},{line,31}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
rabbitmq-server-1    |
rabbitmq-server-1    | Crash dump is being written to: erl_crash.dump...rabbitmq-server-0    | {"init terminating in do_boot",{{badmatch,{error,{not_started,rabbit_common}}},[{rabbit_clusterer,boot,0,[{file,"src/rabbit_clusterer.erl"},{line,31}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
rabbitmq-server-0    | init terminating in do_boot ()
rabbitmq-server-0    |
rabbitmq-server-0    | Crash dump is being written to: erl_crash.dump...rabbitmq-server-1 exited with code 1
rabbitmq-server-0 exited with code 1

And I am using the latest release, where goes wrong?

Support node names as tuples

Does this plugin support long names?

I'm trying to use long names and having the following error:

Ignoring external configuration due to error: "Invalid node: {\"[email protected]\",disc}"

config used:

[{rabbitmq_clusterer,
  [{config,
        [{version, 4},
         {nodes, [
          {"[email protected]", disc},
          {"[email protected]", disc},
          {"[email protected]", disc}]},
          {gospel, {node, "[email protected]"}}
        ]
     }]
}].

Cluster with one node

Hey,

Im trying to automatically create a cluster at startup but I can't get the cluster to start because the first docker container that starts is alone and because of that its a single node in the cluster.config file.

I will keep increasing the config file when more rabbitmqs gets started but I need to be able to start one at first.

Below is the first config that will be created.

[{version, 1},
{nodes, ['[email protected]']},
{gospel, {node, '[email protected]'}
}].

Next container that starts will be

[{version, 1},
{nodes, ['[email protected]', '[email protected]']},
{gospel, {node, '[email protected]'}
}].

PS: dont care about the IP:s I just faked em for show.

The only thing I get from the MQ is "Clusterer stopping node now."

Cheers

rabbit_table:wait_for_replicated/0 is gone in `stable`

This plugin relies on rabbit_table:wait_for_replicated/0 which is gone in stable (upcoming 3.6.7), or more specifically in rabbitmq/rabbitmq-server#1033.

While the goal of rabbitmq/rabbitmq-server#1033 is to make this plugin unnecessary for most, we should either

• Update and produce a new release of this plugin
• Bring that function back (it'd pass false to rabbit_table:wait_for_replicated/1)

The latter should also work for compatibility for master and older/customer builds of this plugin, so we will do that.

No instructions to build *.ez file for the plugin

rabbitmq_server-3.6.15安装失败

[root@VM-66-113 sbin]# ./rabbitmq-server start

BOOT FAILED

Error description:
noproc

Log files (may contain more information):
/data/mq/rabbitmq_server-3.6.15/var/log/rabbitmq/[email protected]
/data/mq/rabbitmq_server-3.6.15/var/log/rabbitmq/[email protected]

Stack trace:
[{gen,do_for_proc,2,[{file,"gen.erl"},{line,228}]},
{gen_event,rpc,2,[{file,"gen_event.erl"},{line,239}]},
{rabbit,ensure_working_log_handlers,0,
[{file,"src/rabbit.erl"},{line,856}]},
{rabbit,'-boot/0-fun-0-',0,[{file,"src/rabbit.erl"},{line,288}]},
{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,424}]},
{init,start_em,1,[]},
{init,do_boot,3,[]}]

=INFO REPORT==== 23-Jul-2018::21:45:15.011802 ===
Error description:
noproc

Log files (may contain more information):
/data/mq/rabbitmq_server-3.6.15/var/log/rabbitmq/[email protected]
/data/mq/rabbitmq_server-3.6.15/var/log/rabbitmq/[email protected]

Stack trace:
[{gen,do_for_proc,2,[{file,"gen.erl"},{line,228}]},
{gen_event,rpc,2,[{file,"gen_event.erl"},{line,239}]},
{rabbit,ensure_working_log_handlers,0,
[{file,"src/rabbit.erl"},{line,856}]},
{rabbit,'-boot/0-fun-0-',0,[{file,"src/rabbit.erl"},{line,288}]},
{rabbit,start_it,1,[{file,"src/rabbit.erl"},{line,424}]},
{init,start_em,1,[]},
{init,do_boot,3,[]}]

{"init terminating in do_boot",noproc}
init terminating in do_boot (noproc)

Crash dump is being written to: erl_crash.dump...done

请问下，这是什么原因呢？

rabbitmqctl "list_vhosts" hangs

I am trying to execute following command but it hangs.
su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "list_vhosts"

When I run command using root user

[root@ssplvm ~]# su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl  "list_vhosts"
{"init terminating in do_boot",{badarg,[{erl_prim_loader,check_file_result,3,[]},{init,get_boot,1,[]},{init,get_boot,2,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ()

When I run command using regular user
It asks for password for rabbitmquser.

Following are details of environment.

=> rabbitmqctl status

[root@ssplvm ~]# rabbitmqctl status
Status of node rabbit@ssplvm ...
[{pid,11605},
 {running_applications,[{rabbit,"RabbitMQ","3.3.5"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {xmerl,"XML parser","1.3.6"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,35733088},
          {connection_procs,2800},
          {queue_procs,5600},
          {plugins,0},
          {other_proc,13419424},
          {mnesia,64168},
          {mgmt_db,0},
          {msg_index,34528},
          {other_ets,763464},
          {binary,16360},
          {code,16707114},
          {atom,602729},
          {other_system,4116901}]},
 {alarms,[]},
 {listeners,[{clustering,25672,"::"},{amqp,5672,"::"}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,764030156},
 {disk_free_limit,50000000},
 {disk_free,16844070912},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,126}]},
 {run_queue,0},
 {uptime,1306}]
...done.

=> rabbitmqctl environment

[root@ssplvm ~]# rabbitmqctl environment
Application environment of node rabbit@ssplvm ...
[{auth_backends,[rabbit_auth_backend_internal]},
 {auth_mechanisms,['PLAIN','AMQPLAIN']},
 {backing_queue_module,rabbit_variable_queue},
 {channel_max,0},
 {channel_operation_timeout,70000},
 {cluster_nodes,{[],disc}},
 {cluster_partition_handling,ignore},
 {collect_statistics,none},
 {collect_statistics_interval,5000},
 {default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
 {default_user,<<"guest">>},
 {default_user_tags,[administrator]},
 {default_vhost,<<"/">>},
 {delegate_count,16},
 {disk_free_limit,50000000},
 {enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
 {error_logger,{file,"/var/log/rabbitmq/[email protected]"}},
 {frame_max,131072},
 {halt_on_upgrade_failure,true},
 {heartbeat,580},
 {hipe_compile,false},
 {hipe_modules,[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
                rabbit_command_assembler,rabbit_framing_amqp_0_9_1,
                rabbit_basic,rabbit_event,lists,queue,priority_queue,
                rabbit_router,rabbit_trace,rabbit_misc,rabbit_binary_parser,
                rabbit_exchange_type_direct,rabbit_guid,rabbit_net,
                rabbit_amqqueue_process,rabbit_variable_queue,
                rabbit_binary_generator,rabbit_writer,delegate,gb_sets,lqueue,
                sets,orddict,rabbit_amqqueue,rabbit_limiter,gb_trees,
                rabbit_queue_index,rabbit_exchange_decorator,gen,dict,ordsets,
                file_handle_cache,rabbit_msg_store,array,
                rabbit_msg_store_ets_index,rabbit_msg_file,
                rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
                mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
                ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,
                ssl]},
 {included_applications,[]},
 {log_levels,[{connection,info}]},
 {loopback_users,[]},
 {msg_store_file_size_limit,16777216},
 {msg_store_index_module,rabbit_msg_store_ets_index},
 {plugins_dir,"/usr/lib/rabbitmq/lib/rabbitmq_server-3.3.5/sbin/../plugins"},
 {plugins_expand_dir,"/var/lib/rabbitmq/mnesia/rabbit@ssplvm-plugins-expand"},
 {queue_index_max_journal_entries,65536},
 {reverse_dns_lookups,false},
 {sasl_error_logger,{file,"/var/log/rabbitmq/[email protected]"}},
 {server_properties,[]},
 {ssl_apps,[asn1,crypto,public_key,ssl]},
 {ssl_cert_login_from,distinguished_name},
 {ssl_listeners,[]},
 {ssl_options,[]},
 {tcp_listen_options,[binary,
                      {packet,raw},
                      {reuseaddr,true},
                      {backlog,128},
                      {nodelay,true},
                      {linger,{true,0}},
                      {exit_on_close,false}]},
 {tcp_listeners,[5672]},
 {trace_vhosts,[]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_high_watermark_paging_ratio,0.5}]
...done.

=> Operating system
CentOS Linux release 7.5.1804 (Core) on VMWare Fusion.

Any help ?

Shipping this Plugin as officially supported

I was just curious about this plugin being shipped as officially supported and in the distributions. I assume there is a very good reason it is not but this plugin could really change the Rabbitmq and Docker landscape with some minor changes to the docker-library/rabbitmq. (I may have already made them and have them ready for a PR) However I am sure there is a reason and just wanted to ask the creators and team members on if this ever could be the case.

Justin P.

Clusterer stopping node now.

I am trying to use rabbitmq_clusterer to cluster with a beforehand known node. In this first node, everything goes smooth and I get the following:

=INFO REPORT==== 17-May-2016::18:37:37 ===
Cluster achieved and Rabbit running.

However, in subsequent nodes (slaves) I get

=INFO REPORT==== 17-May-2016::18:39:40 ===
Clusterer stopping node now.

This is the only logging I ever get from these nodes. After that, rabbitmq-server and rabbitmqctl stop working and the node is lost for good. The hostnames I am providing to the configuration do resolve. Am I overlooking something? In case it matters, I am running inside Docker (Kubernetes).

Detail behavior when stopping a node for example

Hi @michaelklishin,

The README mentions to not execute certain commands but what happen if you stop the service for example?

Thanks in advance!

A working K8s cluster example using clusterer

Hi,

I wanted to offer a more up to date RabbitMQ cluster configuration working on Kubernetes 1.5.X using StatefulSets and this plugin.

It is deploy-ready and all images are public.

You can find it here: https://github.com/nanit/kubernetes-rabbitmq-cluster

Provide Binaries to Download

I'm trying to set up clusterer, and I cannot find a single binary ez file I can download.

rabbit_clusterer:status() returns "Clusterer is booting" after a node was restarted due to pause_minority

When a node had been paused due to pause_minority and later restored rabbit_clustere:status reports "Cluster is booting". Although the cluster is alive and all nodes including this one are part of the cluster. rabbitmqctl cluster_status shows correct output. I looked at the plugin code and looks like it gets zero notification in this case. RabbitMQ restart is controlled by rabbit itself.

How to reproduce:

1. Build a cluster using the plugin and set pause_minority
2. Using iptables drop connectivity of one node to other cluster members and wait until it pauses
3. Remove iptables rules to restore the connectivity
4. Check that rabbitmqctl eval 'rabbit_clusterer:status().' returns incorrect status.

Issue with RabbitMQ 3.7.18

I am trying to use Rabbitmq-clusterer with RabbitMQ 3.7.18. This plugin was working perfectly with RabbitMQ 3.6.5
I am using Rabbitmq-clusterer plugin - 1.0.3

Currently, all nodes are able to create their own cluster. RabbitMQ server is working perfectly on each of node.

Somehow I am not able to setup cluster.
while executing rabbitmqctl eval 'rabbit_clusterer:status().' It gives "Clusterer is pre-booting. []
ok"

With rabbitmqctl eval 'rabbit_clusterer:apply_config().' command, it is giving "{cannot_apply_config_currently,preboot}" output.

Please help if I am missing anything. Please let me know for more information.