cloudwatt / mod_bunny

Distributed Nagios checks with RabbitMQ

mod_bunny: distribute Nagios checks with RabbitMQ

mod_bunny is a Nagios Event Broker (NEB) module that publishes host & service checks through a RabbitMQ AMQP broker, allowing you to distribute the checking load over multiple workers. It is similar to and heavily inspired from Mod-Gearman. The official worker implementation is bunny (written in Go).

Requirements

mod_bunny relies on the following software:

• RabbitMQ C AMQP client library (>= 0.3.0)
• Jansson C JSON library (>= 2.3)

Installation

Currently the installation method a little rough on the edges, since it needs the Nagios source tree to be compiled. The Nagios sources have to be "prepared" because the required config.h header is only available once the ./configure script has been successfully executed (no need to compile the sources). Once the sources are ready to be used, build the module as follows:

Note: you have to use the exact same sources version as the Nagios daemon binary.

NAGIOS_SOURCES=/usr/src/nagios-3.2.3 make

Once compiled, copy the binary module mod_bunny.o to Nagios's modules directory (usually /usr/lib/nagios3/modules).

Configuration

In Nagios main configuration file (usually /etc/nagios3/nagios.cfg), add the following line (replace /usr/lib/nagios3/modules with the exact location of the compiled module on your system):

broker_module=/usr/lib/nagios3/modules/mod_bunny.o /etc/nagios3/mod_bunny.conf

Note: the parameter is the absolute path to mod_bunny's configuration file; not specifying this parameter will make mod_bunny only use its default settings, which will probably won't work for you. Also, if you use multiple broker modules, declare mod_bunny.o first as it needs to intercept some events early.

The configuration file is using JSON format. Here are the supported settings and their default value:

• "host": "localhost" Broker hostname or address
• "port": 5672 Broker port
• "vhost": "/" Broker virtual host
• "user": "guest" Broker account username
• "password": "guest" Broker account password
• "publisher_exchange": "nagios" Broker exchange to connect to for publishing checks messages
• "publisher_exchange_type": "direct" Broker publisher exchange type*
• "publisher_routing_key": "nagios_checks" Routing key to apply when publishing check messages
• "consumer_exchange": "nagios" Broker exchange to connect to for consuming checks result messages
• "consumer_exchange_type": "direct" Broker consumer exchange type
• "consumer_queue": "nagios_results" Queue to bind to for consuming check result messages
• "consumer_binding_key": "nagios_results" Binding key to use to consume check result messages
• "local_hostgroups": [] Hostgroups** for which mod_bunny won't override checks (Nagios-local checks)
• "local_servicegroups": [] Servicegroups** for which mod_bunny won't override checks (Nagios-local checks)
• "hostgroups_routing_table": {} Mapping of AMQP routing keys/hostgroups to use for dispatching host checks
• "servicegroups_routing_table": {} Mapping of AMQP routing keys/servicegroups to use for dispatching service checks
• "retry_wait_time": 3 Time to wait (in seconds) before trying to reconnect to the broker
• "debug_level": 0 Debugging level (0 = none, 1 = show Nagios events and AMQP events, 2 = same as 1 + dump received/sent AMQP messages)

* : To benefit from the round-robin load-balancing RabbitMQ feature, the publisher exchange MUST be of type direct. Read this to understand why.

** : local_hostgroups and local_servicegroups array elements are strings describing shell patterns, e.g. ["*-servers", "nagios_local"]

Basic configuration example:

{
  "host": "some.amqp.broker.example.net",
  "user": "bunny",
  "password": "S3curEP4$$w0rd!"
}

Hostgroups/servicegroups routing table are useful for executing host/service checks by specific bunny workers:

{
  ...

  "hostgroups_routing_table": {
    "nagios_checks_oob": [ "oob", "net-*" ]
  },

  "servicegroups_routing_table": {
    "nagios_checks_www": [ "www" ],
    "nagios_checks_db": [ "mysql", "oracle", "postgresql" ]
  }
}

In the configuration example above, all checks for hosts members of the hostgroup oob and all hostgroups matching the "net-*" wildcard will be published with the routing key "nagios_checks_oob": this way, only bunny workers bound to a queue matching this key will receive the checks. Similarily, all checks for services members of the servicegroup www will be executed by bunny workers bound to a queue matching the routing key "nagios_checks_www". All others host/checks will be published with the routing key defined by the publisher_routing_key setting.

Compatibility

mod_bunny has been tested with Nagios versions 3.2.3, 3.4.1 and 3.5.0 on Ubuntu Linux. Let me know if you successfully made it work on other platforms/versions.

Bugs

Probably. The documentation related to NEB development is almost nonexistent, and the Nagios source code is a nightmare. I heavily relied on Mod-Gearman source code to understand how Nagios internals work, but I might have got or done some things wrong.

Currently mod_bunny only handles host/service checks events. I have no plans to support other events at the moment, but contributions are welcome.

License / Copyright

This software is released under the MIT License.

Copyright (c) 2013 Marc Falzon / Cloudwatt

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

mod_bunny's Issues

Mod_bunny doesn't respect the service_check_timeout

When scheduling service checks and never getting any anwser (with no bunny-worker running for instance), mod_bunny does not apply any service timeout.
In "plain nagios", if service timeout, the service state is set to Critical.

Set a "correlation-id" header in check request AMQP messages

RabbitMQ allows producers to set a correlation-id header (see "Correlation Id" in this tutorial) when sending a message through the broker: it could be helpful to set this to some kind of hash of the properties of each check (e.g. [host, time] for host checks, [host, service, time] for service checks) to allow tracking of a check request and its result when processed by a worker to debug potential problems.

mod_bunny and bunny should support multi amqp broker definition

Hello,

IMHO we should be able to define multiple amqp broker to have a built-in failover method in mod_bunny.
I know we could use haproxy in front of a rabbitmq cluster but I'd prefer to have the smallest infrastructure dependency as possible in my monitoring solution.

Cheers,

Romain

Set a "reply-to" AMQP message header

In order for a bunny worker to process checks from several Nagios servers, the mod_bunny must set a reply-to AMQP message header to tell the worker what routing key to use when publishing the check result back to the right Nagios server.

Trying to compile mod_bunny against nagios 4.4.2

Hi, I am trying to compile mod_bunny against nagios 4.4.2, The dependency modules have been successfully installed.

The output while compiling the code with make.

[root@IRSSRV01 mod_bunny-0.5.0]# NAGIOS_SOURCES=/root/nagios make
cc -std=gnu99 -W -Wall -g -O2 -shared -fPIC -I/root/nagios/include \
	-DNAGIOS_3_5_X=`echo /root/nagios | grep -Ec "nagios[3]?[-_]3\.5"` \
	-o mod_bunny.o \
	mb_hash.c \
	mb_json.c \
	mb_amqp.c \
	mb_thread.c \
	mod_bunny.c \
	-lpthread -lrabbitmq -ljansson
mod_bunny.c:33:21: error: conflicting types for ‘event_broker_options’
 extern int          event_broker_options;
                     ^
In file included from mod_bunny.h:52:0,
                 from mod_bunny.c:23:
/root/nagios/include/nagios.h:170:22: note: previous declaration of ‘event_broker_options’ was here
 extern unsigned long event_broker_options;
                      ^
mod_bunny.c: In function ‘mb_handle_host_check’:
mod_bunny.c:407:5: warning: implicit declaration of function ‘adjust_host_check_attempt_3x’ [-Wimplicit-function-declaration]
     adjust_host_check_attempt_3x(hst, TRUE);
     ^
mod_bunny.c:414:53: error: ‘host’ has no member named ‘host_check_command’
     get_raw_command_line(hst->check_command_ptr, hst->host_check_command, &raw_command, 0);
                                                     ^
mod_bunny.c: In function ‘mb_handle_service_check’:
mod_bunny.c:544:53: error: ‘service’ has no member named ‘service_check_command’
     get_raw_command_line(svc->check_command_ptr, svc->service_check_command, &raw_command, 0);
                                                     ^
mod_bunny.c: In function ‘mb_process_check_result’:
mod_bunny.c:657:5: warning: implicit declaration of function ‘add_check_result_to_list’ [-Wimplicit-function-declaration]
     add_check_result_to_list(cr);
     ^
make: *** [mod_bunny.o] Error 1