isabella232 / puppet-heka

A simple Puppet module for configuring Heka, an event stream processor written in Go

#puppet-heka

github.com/newrelic/puppet-heka

---

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
   • Packages
   • Init scripts and Upstart/systemd files
   • TOML config files
     • Global Heka configuration
     • TOML config files for plugins
3. Setup - The basics of getting started with heka
   • Setup requirements
4. Usage - Configuration options and additional functionality
   • Basic usage
     • Parameter data types
   • Plugins
     • Inputs
       • TcpInput
       • UdpInput
       • StatsdInput
       • StatAccumInput
     • Outputs
       • CarbonOutput
     • Custom plugins
5. Limitations - OS compatibility, etc.
6. Development - Guide for contributing to the module

##Overview

This module installs and configures Heka, a metric/log/event router and processor.

##Module Description

This module installs Heka via the published packages and configures it by templating TOML configuration files for both Heka's global configuration and for plugins.

###Packages

This module uses the RPM packages available on Heka's Github releases page:

https://github.com/mozilla-services/heka/releases

###Init scripts and Upstart/systemd files

Because the Heka packages do not include init scripts or Upstart/systemd unit files, this module includes an Upstart file template for Red Hat/CentOS 6 and a systemd unit file for Red Hat/CentOS 7.

####systemd unit file custom parameters

You can use the $systemd_unit_file_settings parameter in the main heka class to specify any custom systemd unit file key/value config options you want. $systemd_unit_file_settings is a hash and unit file K/V pairs are given as keys/values in the hash:

class { '::heka':
...
  systemd_unit_file_settings => {
    'RestartSec' => '30',
  },
...
}

This parameter is ignored on systems other than Red Hat/CentOS 7.

Note: Keys and values must be strings or they will be ignored and not inserted into the templated unit file.

###TOML config files

####Global Heka configuration

The module templates /etc/heka.toml. Currently, it only specifies a maxprocs value, which you can override with the heka_max_procs parameter:

class { '::heka':
  heka_max_procs         => '2',
}

To specify global configuration options the module doesn't explicitly provide parameters for, you can use the global_config_settings parameter, which is a hash and uses the key/value pairs as the key/value pairs printed in heka.toml:

class { '::heka':
  global_config_settings => {
    'poolsize' => 10000,
  },
}

Heka sets any global config settings that are not specified to its own internal defaults. See Configuring hekad > Global configuration options for more info on what global configuration options are available and what Heka's internal default values are.

####TOML config files for plugins

This module has defined types and and accompanying ERB templates for a few commonly used plugins.

This module also has a more generic defined type and ERB template so you can configure any Heka plugin the module doesn't already include a defined type for.

##Setup

###Setup Requirements

This module requires the following Puppet modules:

• nanlui/staging

##Usage

###Basic usage

The following installs and configures Heka with it's default settings and manages the Heka daemon once everything is installed and configured. The maxprocs in Heka's main config will be set to the value of the processorcount fact. All other global config values won't be printed in the main config file unless specified as parameters. If they aren't printed in the file, Heka will use its own internal default value for each setting.

class { '::heka':}

You can optionally choose to not manage the Heka service:

class { '::heka':
  manage_service => false,
}

Note: The value of this parameter is also used as the default for any plugins configured by the module and will control whether they try to notify the Heka daemon. If the Heka service is not managed by the main heka class, make sure that you don't explicitly set any plugins to notify the daemon with their refresh_heka_service parameter. If you do, catalog compilation will fail, as the Heka service won't be in the catalog for the plugin to send notifications to.

To specify a custom configuration option the module doesn't provide an explicit parameter for, use the global_config_settings parameter:

class { '::heka':
  global_config_settings => {
    'poolsize' => 10000,
    'hostname' => "\"${::fqdn}\"",
  },
}

####Setting cgroup memory limits

Note: This parameter only applies to CentOS 7 systems. If used on CentOS 6 or Debian/Ubuntu systems, it will be ignored.

To set a cgroup memory limit on the Heka daemon, you can use the cgroup_memory_limit parameter. It expects a string of numbers, followed by one of K, M, G or T, for kilobytes, megabytes, gigabytes or terabytes, respectively. Catalog compilation will fail if the value doesn't conform to this regular expression:

\d{1,}[KMGT]

Example:

class { '::heka':
...
  cgroup_memory_limit = '500M',
...
}

####Purging unmanaged TOML configs

You can use the purge_unmanaged_configs parameter to remove any TOML configs not managed by Puppet. The parameter is a boolean and defaults to true:

class { '::heka':
...
  purge_unmanaged_configs => true,
...
}

####Package versioning

To specify a particular Heka package version to install, use the package_download_url and version parameters:

class { '::heka':
  version => '0.9.2',
  package_download_url => 'https://github.com/mozilla-services/heka/releases/download/v0.9.2/heka-0_9_2-linux-amd64.rpm',
}

If you specify an older version than what is currently installed, the module will downgrade the package. However, make sure that the plugins and configuration options you're specifying with the module's plugin types are compatible with the version you are downgrading to!

####Puppet parameter and Heka data types

Booleans

Config values that are supposed to be booleans in a TOML config can be used as a boolean in your Puppet code:

::heka::plugin::input::tcpinput { 'tcpinput1':
...  
  refresh_heka_service => true,
...
}

Strings

Config values that are supposed to be double-quoted strings in the TOML file should be entered as single-quoted strings in your Puppet code:

::heka::plugin::input::tcpinput { 'tcpinput1':
...
  address => '127.0.0.1:5565'
...
}

Note: This only applies to the included defined types already included in the module. If you use the generic heka::plugin type or the global_config_settings parameter of the main heka class, you must enter your value parameters for strings as they will appear in the TOML file. This means including the double-quotes:

::heka::plugin { 'dashboard1':
  type => 'DashboardOutput',
  settings => {
    'address' => '"0.0.0.0:4352"',
...
  },
}

If you need to interpolate Facter facts, enclose the whole parameter in double-quotes and escape the internal set of dobule-quotes that need to be printed:

::heka::plugin { 'dashboard1':
  type => 'DashboardOutput',
  settings => {
    'address' => "\"${::ipaddress_lo}:4352\"",
...
  },
}

Integers

Config values that are supposed to be integers in the TOML config file can be written as integers in your Puppet code:

::heka::plugin { 'dashboard3':
  type => 'DashboardOutput',
  settings => {
...
    'ticker_interval' => 6,
...
  },
}

IP addresses and ports

IP address/port combos in Heka TOML configs are entered as double-quoted strings. You can enter them in your Puppet code as regular strings with single quotes:

::heka::plugin::input::tcpinput { 'tcpinput1':
...
  address => '127.0.0.1:5565'
...
}

You can use Facter facts for IP addresses by using ${} inside of double quotes for variable interpolation in your Puppet code:

::heka::plugin::input::tcpinput { 'tcpinput1':
...
  address => "${::ipaddress_lo}:5565"
...
}

###Plugins

####Common parameters

Each plugin's defined type and the custom plugin type have the following parameters in common:

• plugin_dir: the directory the TOML plugin file will be created in; defaults to /etc/heka.
• plugin_file_name: the name of the TOML plugin file; defaults to $name.toml.
• plugin_file_ensure: the type of file resource to create; defaults to file.
• plugin_file_owner: the owner of the TOML plugin file; defaults to root.
• plugin_file_group: the group of the TOML plugin file; defaults to root.
• plugin_file_mode: the mode of the TOML plugin file; defaults to 0644.
• plugin_file_template: the ERB template to use to generate the TOML plugin file

####Plugin type settings

Heka allows just using the plugin's type as a name if there is only one instance of that plugin that's configured.

Plugins configured via this module, however, require explicit names. The type setting for the plugin is already specified for the plugin types the module includes.

You'll have to specify the type if you use the heka::plugin type to create a custom plugin config.

####Inputs

#####TcpInput

Heka documentation: TcpInput

::heka::plugin::input::tcpinput { 'tcpinput1':
  refresh_heka_service => true,
  address => "${::ipaddress_lo}:5565"
}

#####UdpInput

Heka documentation: UdpInput

::heka::plugin::input::udpinput { 'udpinput1':
  refresh_heka_service => true,
  address => "${::ipaddress_lo}:4484"
}

#####StatsdInput

Heka documentation: UdpInput

Note: An instance of this plugin must be accompanied by an instance of the StatAccumInput in order for Heka to start properly!

::heka::plugin::input::statsdinput { 'statsdinput1':
  address => "0.0.0.0:8125"
}

####StatAccumInput

Heka documentation: StatAccumInput

Note: An instance of this plugin must be used in order for instances of the StatsdInput plugin to work properly!

::heka::plugin::input::stataccuminput { 'stataccuminput1':
  refresh_heka_service => true,
  ticker_interval => 1,
  emit_in_fields => true,
}

####Outputs

#####CarbonOutput

Heka documentation: CarbonOutput

::heka::plugin::output::carbonoutput { 'carbonoutput1':
  address => 'graphiteserver.local:2003',
  message_matcher => "Type == 'heka.statmetric'",
  protocol => 'udp',
}

####Custom plugins

You can use the heka::plugin defined type to generate TOML configs for plugins that the module doesn't already include:

::heka::plugin { 'dashboard3':
  type => 'DashboardOutput',
  settings => {
    'address' => '"10.0.1.120:4354"',
    'ticker_interval' => 6,
  },
}

Note: The type parameter is required.

Settings

settings is a hash that will get used to generate the key/value pairs in the TOML file.

settings => {
  'script_type' => '"lua"',
  'filename' => '"lua_decoders/nginx_access.lua"',
},

...would generate the following in the final TOML file:

filename = "lua_decoders/nginx_access.lua"
script_type = "lua"

Subsection settings

You can use the subsetting_sections parameter to specify any sections of subsettings a plugin might require:

  subsetting_sections => {
    'config' => {
      'log_format' => '\'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"\'',
      'type' => '"nginx.access"',
    },
  }

The subsetting_sections parameter above would generate the following TOML data:

[nginx_access_decoder.config]
log_format = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
type = "nginx.access"

A heka::plugin with a subsetting_sections parameter would look like:

::heka::plugin { 'nginx_access_decoder':
  refresh_heka_service => true,
  type => 'SandboxDecoder',
  settings => {
    'script_type' => '"lua"',
    'filename' => '"lua_decoders/nginx_access.lua"',
  },
  subsetting_sections => {
    'config' => {
      'log_format' => '\'$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"\'',
      'type' => '"nginx.access"',
    },
  }
}

The rendered TOML file the above heka::plugin generates would be:

[nginx_access_decoder]
type = "SandboxDecoder"
filename = "lua_decoders/nginx_access.lua"
script_type = "lua"

  [nginx_access_decoder.config]
  log_format = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
  type = "nginx.access"

##Limitations

Only Red Hat and CentOS are supported right now.

##Development

Fork this repository and create feature branches against the develop branch. Please keep one feature/change/bug fix per feature branch.

###Contribution guidelines

You are welcome to send pull requests to us - however, by doing so you agree that you are granting New Relic a non-exclusive, non-revokable, no-cost license to use the code, algorithms, patents, and ideas in that code in our products if we so choose. You also agree the code is provided as-is and you provide no warranties as to its fitness or correctness for any purpose.