GithubHelp home page GithubHelp logo

example42 / puppet-splunk Goto Github PK

View Code? Open in Web Editor NEW
12.0 9.0 29.0 99 KB

Splunk Puppet Module

Home Page: http://www.example42.com

License: Other

Ruby 25.25% Puppet 64.27% Shell 4.95% HTML 5.53%

puppet-splunk's Introduction

Puppet module: splunk

DEPRECATION NOTICE

This module is no more actively maintained and will hardly be updated.

Please find an alternative module from other authors or consider Tiny Puppet as replacement.

If you want to maintain this module, contact Alessandro Franceschi

This is a Puppet splunk module from the second generation of Example42 Puppet Modules.

Made by Alessandro Franceschi / Lab42 - http://www.example42.com

Released under the terms of Apache 2 License.

Check Modulefile for dependencies.

##= GENERAL USAGE This module installs splunk server or splunk universal forwarder. It uses packages that can be retrieved from http://www.splunk.com. It has been tested on version 4.3 but should work also on earlier releases.

Note that you have to specify a package_source if you don't have the relevant packages in your repo(s).

To install Splunk server (searcher, indexer, web interface) via yum/apt:

    class { "splunk":
      install => "server",
    }

To install Universal Forwarder via yum/apt:

    class { "splunk":
      install        => "forwarder", # This is the default, can be omitted 
      forward_server => [ "splunk.example42.com:9997", ], # Must be an array; use host:port format
    }

To install the Splunk server from a custom url

    class { "splunk":
      install        => "server",
      install_source => "http://files.example42.com/splunk-4.3-115073-linux-2.6-x86_64.rpm",
      admin_password => "mypassword",
    }

To install the Universal Forwarder from a custom url, setting an admin password and custom local files to forward:

    class { "splunk":
      # install        => "forwarder", # This is the default, can be omitted 
      install_source => "http://files.example42.com/splunkforwarder-4.3-115073-linux-2.6-amd64.deb",
      forward_server => [ "splunk.example42.com:9997", ], # Must be an array; use host:port format
      monitor_path   => [ "/var/log/tomcat6/catalina.out" , "/var/log/apache2/" ],
      admin_password => "!what4wONDErFu!P4ssw0rd",
    }

The variables used in the module can be passed in 2 ways:

  • With the old style "Set variables and include class" pattern:

      $splunk_forward_server = [ "splunk.example42.com:9997", ],
      include splunk
    
  • As a parametrized class:

      class { "splunk":
        forward_server => [ "splunk.example42.com:9997", ],
      }
    

You can even, under some degrees, mix these two patterns.

You can for example set general top scope variables that affect all your parametrized classes:

    $puppi = true
    $monitor = true
    $monitor_tool = [ "nagios" , "munin" , "puppi" ]
    class { "splunk":
      forward_server => [ "splunk.example42.com:9997", ],
    }

The above example has the same effect of:

    class { "splunk":
      forward_server => [ "splunk.example42.com:9997", ],
      puppi          => true,
      monitor        => true,
      monitor_tool   => [ "nagios" , "munin" , "puppi" ],
    }

Note that if you use the "Set variables and include class" pattern you can define variables only at the top level scope or in a ENC (External Node Classifer) like Puppet Dashboard, Puppet Enterprise Console or The Foreman.

Below you have an overview of the most important module's parameters (you can mix and aggregate them).

The examples use parametrized classes, but for all the parameters you can set a $splunk_ top scope variable.

For example, the variable "$splunk_absent" is equivant to the "absent =>" parameter.

##= USAGE - Basic management

  • Install splunk with default settings

      class { "splunk": }
    
  • Disable splunk service.

      class { "splunk":
        disable => true,
      }
    
  • Disable splunk service at boot time, but don't stop if is running.

      class { "splunk":
        disableboot => true,
      }
    
  • Remove splunk package

      class { "splunk":
        absent => true,
      }
    
  • Enable auditing without without making changes on existing splunk configuration files

      class { "splunk":
        audit_only => true,
      }
    

##= USAGE - Overrides and Customizations

  • You can set custom sources for some commonly used splunk configuration files Note that if you do this, the omonimous existing files ( in $splunkdir/etc/system/local ) are removed. So, for example, the name of the forward server set with $forward_server is ignored if you use a custom template_server

      class { "splunk":
        template_inputs  => "example42/splunk/inputs.conf.erb",
        template_outputs => "example42/splunk/outputs.conf.erb",
        template_server  => "example42/splunk/server.conf.erb",
        template_web     => "example42/splunk/web.conf.erb",
      }
    
  • Use custom source directory for the whole configuration dir Also for this case existing configuration might be overwritten

      class { "splunk":
        source_dir       => "puppet:///modules/example42/splunk/conf/",
        source_dir_purge => false, # Set to true to purge any existing file not present in $source_dir
      }
    
  • Define custom options that can be used in the custom templates without the need to add parameters to the splunk class

      class { "splunk":
        template_inputs => "example42/splunk/inputs.conf.erb",
        options  => {
          'fschange' => [ "/etc" , "/usr/local/etc" ]
          'monitor'  => [ "/var/log/apache/error.log" , "/var/log/tomcat/catalina.out" ],
        },
      }
    
  • Automaticallly include a custom subclass

      class { "splunk:"
        my_class => 'splunk::example42',
      }
    

##= USAGE - Example42 extensions management

  • Activate puppi (recommended, but disabled by default) Note that this option requires the usage of Example42 puppi module

      class { "splunk": 
        puppi    => true,
      }
    
  • Activate puppi and use a custom puppi_helper template (to be provided separately with a puppi::helper define ) to customize the output of puppi commands

      class { "splunk":
        puppi        => true,
        puppi_helper => "myhelper", 
      }
    
  • Activate automatic monitoring (recommended, but disabled by default) This option requires the usage of Example42 monitor and relevant monitor tools modules

      class { "splunk":
        monitor      => true,
        monitor_tool => [ "nagios" , "monit" , "munin" ],
      }
    
  • Activate automatic firewalling This option requires the usage of Example42 firewall and relevant firewall tools modules

      class { "splunk":       
        firewall      => true,
        firewall_tool => "iptables",
        firewall_src  => "10.42.0.0/24",
        firewall_dst  => "$ipaddress_eth0",
      }
    

Build Status

puppet-splunk's People

Contributors

adrianlzt avatar alvagante avatar claytono avatar davids avatar dwagon avatar evanstachowiak avatar matthewfischer avatar pjelar avatar rickvdp avatar velothump avatar yasn77 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-splunk's Issues

Use yum for custom installs

Could you switch to yum not rpm for custom installs? Then yum won't report package database altered outside of yum.

yum install

example42-firewall as a dependency

Can this be safely & forcefully uninstalled if this feature is not being used?

This conflicts with puppetlabs-firewall and we are not using the firewall features provided by example42-splunk.

Deprecation warning regarding variable access

When I run puppet while using the puppet-splunk module, included in the puppet output to stderr is this:

Warning: Variable access via 'monitor_path' is deprecated. Use '@monitor_path' instead. template[/etc/puppet/modules/splunk/templates/add_monitor.erb]:10 (at /etc/puppet/modules/splunk/templates/add_monitor.erb:10:inresult')`

Adding the @ character should fix.

apt failing to locate splunkforwarder package when install_source is provided

I'm including an install_source parameter to the Splunk site to grab the debian package. The puppet_manage_package file is created with the wget statement, the /tmp/splunk.deb file is downloaded successfully, but the Puppet run fails because apt cannot locate the package 'splunkforwarder'.

Here is a gist of all of my related Puppet files including site.pp, the example42-splunk init.pp, the run output, and the specific error(s) related to the splunk module.

As a side note, puppi is loaded from the modulepath.

https://gist.github.com/bmurt/1b6bd2540deec015324b

Issue when running the splunk module for the first time

Hi Aleassandro,

I am running your puppet module for splunk and I am getting an error related to some misconfiguration on the ERB template "add_forward_server.erb". The error message I am getting when running the puppet agent is the following:

?Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to parse template splunk/add_forward_server.erb:
Filepath: /etc/puppetlabs/puppet/modules/splunk/templates/add_forward_server.erb
Line: 5
Detail: undefined method `each' for "lxexhindladev81.pexa:9997":String
at /etc/puppetlabs/puppet/modules/splunk/manifests/init.pp:397 on node bat02-shr.pexa
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run"

My site.pp is the following:

class { "splunk" :
install => "forwarder",
forward_server => "lxexhindladev81.pexa:9997",
install_source => "/pexa/SOE_applicator/general/splunk/splunk-5.0.1-143156-Linux-x86_64.tgz",
disable => false,
}

The ERB template content is:

!/bin/sh

File Managed by Puppet

Adding forward servers

<% @forward_server.each do |fs| -%>
<%= scope.lookupvar('splunk::basedir') %>/bin/splunk add forward-server <%= fs %> --accept-license --answer-yes --auto-ports --no-prompt -auth admin:<%= scope.lookupvar('splunk::admin_password') %>
<% end -%>

Service restart (done here to skip Puppet dependencies hell)

/etc/init.d/splunk restart

Please let me know if you need any further information. I can provide you with the manifest files if you require.

Thanks,

Rafael

Hack to automatic accept license

This command
"${splunk::basedir}/bin/splunk enable boot-start --accept-license --answer-yes --no-prompt"
Generates init.d/splunk like this:
"/opt/splunk/bin/splunk" start --no-prompt --answer-yes

I have to put this exec in my manifest to workaround that:

exec { 'splunk-accept-license':
  command => '/bin/sed -i "s/--answer-yes/--accept-license --answer-yes/" /etc/init.d/splunk',
  require => Package['splunk'],
  before => Service['splunk'],
}

Splunk not added to chkconfig

Hi,

Is there anything extra I need to add in order to get splunk added to chkconfig?

err: /Stage[main]/Splunk/Service[splunk]/ensure: change from stopped to running failed: Could not enable splunk: Execution of '/sbin/chkconfig splunk on' returned 1: service splunk does not support chkconfig

Splunk Universal Forwarder service name incorrect in 7.2.x on new installs

Expected Behavior

Puppet can correctly start the splunk service

Actual Behavior

puppet has failed attempt to start the splunk forwarder service on every run

Steps to Reproduce the Problem

clean install new system, include the splunk class with a 7.2.x version package (i have tested 7.2.6)

The main issue is that on current systems (Ubuntu 18.04 and CentOS > 7) using systemd the service file is actually created as: SplunkForwarder.service (and does appear to be case sensitive)

and it would appear that the service definition in init.pp is just calling the service name as splunk so this would need to be dynamically determined or we must manage the actual service file fully

Specifications

Please add this info:

  1. Facter data
os => {
  architecture => "amd64",
  distro => {
    codename => "bionic",
    description => "Ubuntu 18.04.2 LTS",
    id => "Ubuntu",
    release => {
      full => "18.04",
      major => "18.04"
    }
  },
  family => "Debian",
  hardware => "x86_64",
  name => "Ubuntu",
  release => {
    full => "18.04",
    major => "18.04"
  },
  selinux => {
    enabled => false
  }
}
  1. Version of Puppet and of the module -

5.5.14 - running on master
3. The relevant Puppet code and eventually Hiera data

	# THIS IS WIP to specifically move stuff to repo based install sources - the above code block will eventually disappear as will this
	case $operatingsystem {
		'ubuntu', 'debian': { include profiles::splunk::repo }
      	'centos', 'redhat': { #NO OP TO NOT FAIL - repo is done in base YUM config already 
		}
		default: {  fail('Unrecognized operating system for Splunk Install') }
	}
	$install_source = ''

    class { 'splunk':
        install           => 'forwarder',
        deployment_server => $deployment_server,
        admin_password    => 'watchingthepuppet',
        install_source    => $install_source,
    }

RSpec code coverage less than 100%

Expected Behavior

  • RSpec Code Coverage at 100%

Actual Behavior

  • RSpec Code Coverage is less than 100%

Steps to Reproduce the Problem

  1. run RSpec code coverage test

Allow non-http downloads

Is there a way to specify a local file or yum repository? I don't see it anywhere in the docs and it fails with install_source => '/opt/splunk.rpm'.

Puppetforge and GitHub differences

I recently installed this module via puppet forge and have noticed that it is different from the contents provided here. In particular, the version from puppet forge contains the spec/fixtures directory.

There are a number of problems with this:

invalid symlink:

spec/fixtures/modules/splunk -> /Users/al/Documents/github/EXAMPLE42/puppet-modules/splunk

presence of .git directories:

./spec/fixtures/modules/concat/.git
./spec/fixtures/modules/firewall/.git
./spec/fixtures/modules/iptables/.git
./spec/fixtures/modules/puppi/.git
./spec/fixtures/modules/monitor/.git

Is the fixtures directory needed?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.