damm / chef-logstash

don't use this!

Description

This is an fork of the semi-official 'all-in-one' Cookbook. It has been reduced to a simple installation method and a resource and provider to configure Logstash.

Requirements

Review the Berksfile, currently this cookbook is not exportable as is.

Attributes

Note only the attributes used are listed below.

logstash::default

Key	Type	Description	Default
['logstash']['basedir']	String	Base Directory where logstash is installed	/opt/logstash
['logstash']['user']	String	Username to run logstash as	logstash
['logstash']['group']	String	Group Name to run logstash as	logstash
['logstash']['join_groups']	Array	An array of Operative System groups to join. Usefull to gain read privileges on some logfiles.	[]
['logstash']['create_account']	[TrueClass, FalseClass]	Create account, info from `user` and `group` is used.	true

logstash::server

Key	Type	Description	Default
['logstash']['server']['install_method']	String	How to install logstash? either source or jar	jar
['logstash']['server']['version']	String	The version of Logstash to install. Only applies to jar installation method.	1.1.12
['logstash']['server']['source_url']	String	The URL of the Logstash jar to download.	https://logstash.objects.dreamhost.com/release/logstash-1.1.12-flatjar.jar
['logstash']['server']['checksum']	String	The checksum of the jar file. Only applies to the jar install method.	e75bce7c88461116fbd2c7c473d8c8999c152ab6c618caa58b3d0d88feeb77fd
['logstash']['server']['xmx']	String	The maximum memory to assign to the JVM.	1024M
['logstash']['server']['java_opts']	String	Additional params you want to pass to the JVM
['logstash']['server']['gc_opts']	String	Specify your garbage collection options to pass to the JVM	-XX:+UseParallelOldGC
['logstash']['server']['ipv4_only']	[TrueClass, FalseClass]	Prefer the IPv4 stack?	false
['logstash']['server']['debug']	[TrueClass, FalseClass]	Run logstash with -v option?	false

logstash::source

Key	Type	Description	Default
['logstash'][['source']['repo']	String	The git repo to use for the logstash source code of Logstash	git://github.com/logstash/logstash.git
['logstash']['source']['sha']	String	The sha/branch/tag of the repo you wish to clone. Uses node['logstash']['server']['version'] by default.	nil
['logstash']['source']['java_home']	String	your JAVA_HOME location. Needed explicity for ant when building JRuby	/usr/lib/jvm/java-6-openjdk/

logstash::index_cleaner

Key	Type	Description	Default
['logstash']['index_cleaner']['days_to_keep']	Fixnum	number of days from today of Logstash index to keep.	31
['logstash']['index_cleaner']['cron']['minute']	String	Minute to run the index_clleaner cron job	0
['logstash']['index_cleaner']['hour']	String	Hour to run the index_cleaner cron job	*
['logstash']['index_cleaner']['cron']['log_file']	String	Path to direct the index_Cleaners cron job's stdout and stderr to	/dev/null

Resources and Providers

This cookbook provides an resource and an corresponding provider.

indexer.rb

Actions:

• create - create a logstash indexer

Attribute Parameters:

logstash::indexer

Key	Type	Description	Default
basedir	String	Base path of where to install logstash to	/opt/logstash
user	String	Default username logstash runs as	logstash
group	String	Group that the logstash user belongs to	logstash
log_dir	String	Log directory to store logstash console logs	/var/log/logstash
pid_dir	String	/var/run/logstash	/var/run/logstash
patterns_dir	String	Directory where to store the pattern files at	new_resource.basedir + "server/etc/patterns"
base_config	String	The name of the template to use for generating server configuration files	server.conf.erb
base_config_cookbook	String	Specifies where to find the template to use for the template specified in base_config	logstash
xms	String	The minimum memory to assign the JVM.	1024M
xmx	String	The maximum memory to assign the JVM.	1024M
java_opts	String	Additional params you want to pass to the JVM	String.new
gc_opts	String	Specify your garbage collection options to pass to the JVM	-XX:+UseParallelOldGC
ipv4_only	[TrueClass, FalseClass]	Add jvm option preferIPv4Stack?	false
filters	Array	Specifies what filters should be used in the indexer
inputs	Array	Specifies what inputs should be used in the indexer
patterns	Array	Specifies what grok and multiline filters should be made availableto the indexer
output	Array	Specifies where to output and how

Usage

include_recipe "redis"  
include_recipe "rabbitmq::default"  
  logstash_indexer "syslog" do  
    inputs([  
      :redis => {  
      :type => "redis-int",  
      :host => "127.0.0.1",  
      :data_type => "list",  
      :key => "systemlogs:beaver",  
      :format => "json_event"  
      }])  
  patterns([  
      :linuxsyslog => {  
      :SYSLOGBASE2 => "(?:%{SYSLOGTIMESTAMP:timestamp}|%{TIMESTAMP_ISO8601:timestamp8601}) (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:",  
      :SYSLOGPAMSESSION => "%{SYSLOGBASE} (?=%{GREEDYDATA:message})%{WORD:pam_module}\(%{DATA:pam_caller}\): session %{WORD:pam_session_state} for user %{USERNAME:username}(?: by %{GREEDYDATA:pam_by})?",  
      :CRON_ACTION => "[A-Z ]+",  
      :CRONLOG => "%{SYSLOGBASE} \(%{USER:user}\) %{CRON_ACTION:action} \(%{DATA:message}\)",  
      :SYSLOGLINE => "%{SYSLOGBASE2} %{GREEDYDATA:message}",  
      # IETF 5424 syslog(8) format (see http://www.rfc-editor.org/info/rfc5424)  
      :SYSLOG5424PRI => "(?:\<%{NONNEGINT}\>)",  
      :SYSLOG5424SD => "(?:\[%{DATA}\]+|-)",  
      :SYSLOG5424LINE => "%{SYSLOG5424PRI:syslog5424_pri}%{NONNEGINT:syslog5424_ver} (%{TIMESTAMP_ISO8601:syslog5424_ts}|-) (%{HOSTNAME:syslog5424_host}|-) (%{WORD:syslog5424_app}|-) (%{WORD:syslog5424_proc}|-) (%{WORD:syslog5424_msgid}|-) %{SYSLOG5424SD:syslog5424_sd} %{GREEDYDATA:syslog5424_msg}"  
      }])  
  filters([  
      :grok => {  
      :type => "linuxsyslog",  
      :pattern => "%{SYSLOGLINE}"  
      }])  
  output([  
      :elasticsearch_http => {  
      :host => "localhost",  
      :port => 9200,  
      :index => "systemlogs-%{+YYYY.MM.dd}",  
      :flush_size => 1  
      },  
      :graphite => {  
      :host => "localhost",   
      :port => 2013,  
      :metrics => ["logstash.systemlogs.events", "1"]  
      }])  
  action :create  
end

Source vs. Jar install methods

Both agent and server support an attribute for how to install. By default this is set to jar to use the 1.1.1preview as it is required to use elasticsearch 0.19.4. The current release is defined in attributes if you choose to go the source route.

Suggested Usage

knife cookbook create orgname_logstash

• Ensure this cookbook depends on this logstash cookbook ruby depends "logstash"

• Determine how you want to implement your settings, load them in a recipe like in the usage above or load the data in a data bag; and create a recipe to load said data bag to consume and create your logstash instances

No implementation details are provided for configuring RabbitMQ, Redis or any input methods. Nor are there any limitations in the configuration

• Determine how you want to ship logs to logstash?

Other cookbooks should provide shippers to logstash. This cookbook aims only support the logstash server.

License and Author

• Author: Scott M. Likens
• Author: John E. Vincent
• Author: Bryan W. Berry ([email protected])
• Author: Richard Clamp (@richardc)
• Author: Juanje Ojeda (@juanje)
• Author: @benattar
• Copyright: 2013, Scott M. Likens
• Copyright: 2012, John E. Vincent
• Copyright: 2012, Bryan W. Berry
• Copyright: 2012, Richard Clamp
• Copyright: 2012, Juanje Ojeda
• Copyright: 2012, @benattar

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.