GithubHelp home page GithubHelp logo

graylog2 / puppet-graylog Goto Github PK

View Code? Open in Web Editor NEW
9.0 17.0 27.0 107 KB

Install and configure a Graylog system via Puppet

License: Apache License 2.0

Ruby 44.97% Puppet 53.27% HTML 0.62% Dockerfile 1.13%
graylog puppet puppet-module puppet-forge log-management log-analysis

puppet-graylog's Introduction

Graylog Puppet Module

Build Status Puppet Forge Puppet Forge - downloads

Table of Contents

  1. Description
  2. Setup - The basics of getting started with graylog
  3. Usage - Configuration options and additional functionality
  4. Reference - An under-the-hood peek at what the module is doing and how
  5. Limitations - OS compatibility, etc.
  6. Development - Guide for contributing to the module

Description

This module can be used to install and configure a Graylog system. (https://www.graylog.org/)

Native Types

Native types to configure dashboards, inputs, streams and others are provided by the community maintained puppet-graylog_api module.

Setup

What graylog affects

The graylog module manages the following things:

  • APT/YUM repository
  • Graylog packages
  • Graylog configuration files
  • Graylog service

Setup Requirements

The module only manages Graylog itself. You need other modules to install the required dependencies like MongoDB and OpenSearch.

You could use the following modules to install dependencies:

Beginning with graylog

The following modules are required to use the graylog module:

Those dependencies are automatically installed if you are using the Puppet module tool or something like librarian-puppet.

Puppet Module Tool

Use the following command to install the graylog module via the Puppet module tool.

puppet module install graylog/graylog

librarian-puppet

Add the following snippet to your Puppetfile.

mod 'graylog/graylog', 'x.x.x'

Make sure to use the latest version of the graylog module!

Usage

As mentioned above, the graylog module only manages the Graylog system. Other requirements like MongoDB and OpenSearch need to be managed via other modules.

The following config creates a setup with MongoDB, OpenSearch and Graylog on a single node.

class { 'mongodb::globals':
  manage_package_repo => true,
}->
class { 'mongodb::server':
  bind_ip => ['127.0.0.1'],
}

class { 'opensearch':
  version => '2.9.0',
}

class { 'graylog::repository':
  version => '5.1'
}->
class { 'graylog::server':
  package_version => '5.1.0-6',
  config          => {
    'password_secret' => '...',    # Fill in your password secret
    'root_password_sha2' => '...', # Fill in your root password hash
  }
}

A more complex example

class { '::graylog::repository':
  version => '5.1'
}->
class { '::graylog::server':
  config  => {
    is_leader                                          => true,
    node_id_file                                       => '/etc/graylog/server/node-id',
    password_secret                                    => 'password_secret',
    root_username                                      => 'admin',
    root_password_sha2                                 => 'root_password_sha2',
    root_timezone                                      => 'Europe/Berlin',
    allow_leading_wildcard_searches                    => true,
    allow_highlighting                                 => true,
    http_bind_address                                  => '0.0.0.0:9000',
    http_external_uri                                  => 'https://graylog01.domain.local:9000/',
    http_enable_tls                                    => true,
    http_tls_cert_file                                 => '/etc/ssl/graylog/graylog_cert_chain.crt',
    http_tls_key_file                                  => '/etc/ssl/graylog/graylog_key_pkcs8.pem',
    http_tls_key_password                              => 'sslkey-password',
    rotation_strategy                                  => 'time',
    retention_strategy                                 => 'delete',
    elasticsearch_max_time_per_index                   => '1d',
    elasticsearch_max_number_of_indices                => '30',
    elasticsearch_shards                               => '4',
    elasticsearch_replicas                             => '1',
    elasticsearch_index_prefix                         => 'graylog',
    elasticsearch_hosts                                => 'http://opensearch01.domain.local:9200,http://opensearch02.domain.local:9200',
    mongodb_uri                                        => 'mongodb://mongouser:[email protected]:27017,mongodb02.domain.local:27017,mongodb03.domain.local:27017/graylog',
  },
}

Reference

Classes

Public Classes

  • graylog::repository: Manages the official Graylog package repository
  • graylog::server: Installs, configures and manages the Graylog server service
  • graylog::allinone: Creates a full Graylog setup including MongoDB and OpenSearch

Private Classes

  • graylog::params: Default settings for different platforms
  • graylog::repository::apt: Manages APT repositories
  • graylog::repository::yum: Manages YUM repositories

Class: graylog::repository

version

This setting is used to set the repository version that should be used to install the Graylog package. The Graylog package repositories are separated by major version.

It defaults to $graylog::params::major_version.

Example: version => '5.1'

url

This setting is used to set the package repository url.

Note: The module automatically detects the url for your platform so this setting should not be changed.

proxy

This setting is used to facilitate package installation with proxy.

release

This setting is used to set the package repository release.

Note: The Graylog package repositories only use stable as a release so this setting should not be changed.

Class: graylog::server

The graylog::server class configures the Graylog server service.

package_name

This setting is used to choose the Graylog package name. It defaults to graylog-server to install Graylog Open. You can use graylog-enterprise to install the Graylog Enterprise package.

Example: package_name => 'graylog-server'

package_version

This setting is used to choose the Graylog package version. It defaults to installed which means it installs the latest version that is available at install time. You can also use latest so it will always update to the latest stable version if a new one is available.

Example: package_version => '5.1.0-6'

config

This setting is used to specify the Graylog server configuration. The server configuration consists of key value pairs. Every available config option can be used here.

See the example graylog.conf to see a list of available options.

Required settings:

  • password_secret
  • root_password_sha2

Please find some default settings in $graylog::params::default_config.

Example:

config => {
  'password_secret'    => '...',
  'root_password_sha2' => '...',
  'is_leader'          => true,
  'output_batch_size'  => 2500,
}
user

This setting is used to specify the owner for files and directories.

Note: This defaults to graylog because the official Graylog package uses that account to run the server. Only change it if you know what you are doing.

group

This setting is used to specify the group for files and directories.

Note: This defaults to graylog because the official Graylog package uses that account to run the server. Only change it if you know what you are doing.

ensure

This setting is used to configure if the Graylog service should be running or not. It defaults to running.

Available options: running, 'stopped'

enable

This setting is used to configure if the Graylog service should be enabled. It defaults to true.

java_initial_heap_size

Sets the initial Java heap size (-Xms) for Graylog. Defaults to 1g.

java_max_heap_size

Sets the maximum Java heap size (-Xmx) for Graylog. Defaults to 1g.

java_opts

Additional java options for Graylog. Defaults to ``.

restart_on_package_upgrade

This setting restarts the graylog-server service if the os package is upgraded. It defaults to false.

Class: graylog::allinone

The graylog::allinone class configures a complete Graylog system including MongoDB and OpenSearch

Note: This is nice to quickly setup a running system on a single node but should only be used for testing or really small setups. Use the graylog::server class for production deployments.

This class is using two Puppet modules from the forge to setup a complete system. Please make sure you have these installed before using the graylog::allinone class.

Requirements:

opensearch

This setting is used to configure the opensearch Puppet module.

There is only on possible hash key:

  • version: The OpenSearch version to use

Example:

opensearch => {
  version => '2.9.0',
}
graylog

This setting is used to configure the graylog::repository and graylog::server classes.

Example:

graylog => {
  major_version => '5.1',
  config        => {
    # ... see graylog::server description for details
  },
}

Limitations

Supported Graylog versions:

  • 5.x

Supported platforms:

  • Ubuntu/Debian
  • RedHat/CentOS

Development

You can test this module by using Vagrant. It uses the graylog::allinone class to setup a complete system inside the Vagrant box.

$ vagrant up rockylinux8
$ vagrant provision rockylinux8

This is a quick way to see how the module behaves on a real machine.

Please see the CONTRIBUTING.md and CODE_OF_CONDUCT.md files for further details.

Release New Version

  1. Update and commit CHANGELOG
  2. Bump version via bundle exec rake -f Rakefile.release module:bump:minor (or major/patch)
  3. Commit metadata.json
  4. Test build with bundle exec rake -f Rakefile.release module:build
  5. Tag release with bundle exec rake -f Rakefile.release module:tag
  6. Push release to PuppetForge with bundle exec -f Rakefile.release rake module:push
  7. Push commits and tags to GitHub with git push --follow-tags

puppet-graylog's People

Contributors

bernd avatar clxnetom avatar cruelsmith avatar dependabot[bot] avatar djschaap avatar h0tw1r3 avatar identw avatar jalogisch avatar jsimpso avatar k2patel avatar mhutter avatar nomoresecrets avatar timdeluxe avatar towo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppet-graylog's Issues

Please allow for puppetlabs-apt 6.x

puppetlabs-apt is currently at version 6.1.1.

Please allow using puppet-graylog with that series (currently limited to < 5.0.0).

Many thanks!

Gentoo & Sabayon support.

https://forge.puppet.com/gentoo/portage

https://packages.gentoo.org/.
app-admin/graylog2,

Energe -bkv app-admin/greylog2
(Build, use or fetch package/s if prebuilt , verbose)

Can control via above puppet models Gentoo Portage

Gentoo family, includes, calculate , Redcore, ChromeOS, Sabayon, Coreos... Funtoo ,Clover , etc.

https://forge.puppet.com/optiz0r/sabayon
Takes care of entropy Package Management
(Sabayon is Derived of Gentoo)
(Equo up , equo u , equo I app-admin/greylog2 .)
Though emerge Portage is useable , it can muck up a Sabayon Instal at times.

Please ignore.

--Sorry I missed the part about installing elasticsearch, I am editing and closing this. --

Graylog 5 has its own Java

For Ubuntu-installs:

This module modifies /etc/default/java and inserts the environment variable JAVA to point to /usr/bin/java.

As of version 5 Java is included and this variable should not be set unless needed.

The workaround for this until it's fixed is to either install supported version of java och create a link to for /usr/bin/java to /usr/share/graylog-server/jvm/bin/java.

Use `length()` rather than `size()`

The size() function in puppet's stdlib is deprecated; more importantly, if the password hash begins with numerals, followed by e, followed by some more numerals (and then eventually more letters), the size() function will think it's a float in scientific notation and complain (at least on the JRuby-based puppetserver implementation).

Implementation of size checks type of input by calling Ruby's Float() method: https://github.com/puppetlabs/puppetlabs-stdlib/blob/ef267080d65b94f4e2710460d2abe968eaa0d1aa/lib/puppet/parser/functions/size.rb#L27

Float() method on JRuby thinks 5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8 is a Float (specificially, it reads it as Infinity; presumably it gets as far as 5e884, sees that this is already bigger than Float::MAX, and stops parsing).

I'd have filed this as a bug with the stdlib, except they already deprecated size. length has a much saner implementation.

Misc Config Updates

Just a list of a few things that need updating:

In the Elasticsearch config:

  • We should set oss => true so the OSS version is installed.
  • The very important action.auto_create_index setting is not in there for Elasticsearch.

The version of Graylog is set to 3.0. We should update this to the latest version.

These warnings appear during the install:

Warning: Module 'puppetlabs-apt' (v7.4.2) fails to meet some dependencies:
  'graylog-graylog' (v0.9.0) requires 'puppetlabs-apt' (>= 2.2.2 < 7.0.0)
Warning: Module 'puppetlabs-stdlib' (v6.3.0) fails to meet some dependencies:
  'graylog-graylog' (v0.9.0) requires 'puppetlabs-stdlib' (>= 4.16.0 < 6.0.0)

Why is show_diff forced to true?

show_diff => true,

First, the File[] resource in Puppet defaults to true here. Second, by forcing this to true, you make it difficult for us to disable this. The server.conf file includes unencrypted secrets (password_secret) that we do not want to see in our log files when we run Puppet.

service restart on package upgrade

Is there a particular reason why the service is not restarted (or an option is present that let's you restart the service) when the package is upgraded/changed?

Maybe something like

  package { 'graylog-server':
    ensure => $package_version,
    notify   => Service['graylog-server'],
  }

or

Boolean $restart_on_upgrade => false,

[...]

$package_options = $restart_on_upgrade ? {
  true    => { notify => Service['graylog-server'] },
  default => {},    
}

package { 'graylog-server':
  ensure => $package_version,
  *      => $package_options,
}

Apt repo class does not work properly

The apt repo class doesn't work if /etc/apt/apt.conf.d/01proxy is not present at all.

Error: /Stage[main]/Graylog::Repository::Apt/File_line[Remove graylog config from apt proxy file]: Could not evaluate: No such file or directory @ rb_sysopen - /etc/apt/apt.conf.d/01proxy

Workaround for now is to create an empty file but that's a bit ugly.
Workaround doesn't work either. Apt module removes the file if no proxy is configured.

changing "graylog::repository -> $version" has no effect

Hi,

I'm using v0.2.1 of this module (from forge). Looks like "graylog::repository -> $version" has no effect. Changing the value from "2.1" to "2.2" does not change the repo url in file /etc/yum.repos.d/graylog.repo.

Edit: same issue at the master branch

Cheers

Permissions on /etc/graylog/server are wrong

While the server.conf belongs to user graylog, the directory itself belongs to root.

This leads to the server not being able to create a node-id file, which is usually related to connectivity issues to elasticsearch.

Chown'ing the server directory to graylog fixed the issue.

PDK usage

Please migrate to PDK or update metadata.json regarding support OS/Puppet Versions.

'Unknown function length'

I'm at a loss as to why this isn't working.

First thing i should point out, on forge.puppet.com, the stdlib version required is listed as 4.11. We were on 4.14, so i thought we were good to go; however, i got this error:
Error: Unknown function length at /vagrant/modules/graylog/manifests/server.pp:23

A little digging, and i see that the length function was added in stdlib 4.16.

Okay, i'll update stdlib. Well, that's been done - puppet module list shows puppetlabs-stdlib (v4.24.0). And yet i'm still getting the same Unknown function length error.

We're on puppet 3.8.5, the latest supported in Ubuntu 16.04 LTS.

Feature Suggestion: Add capability for installation behind proxy (yum)

It would be great if the puppet module for Graylog supported the option to specify a proxy to be written into the yum repo file, as Elasticsearch and MongoDB do.

I'll happily submit a pull request to enable this functionality - it's the only thing breaking my setup flow at the moment!

Cheers

Graylog 4.3 deprecates configuration is_master

Starting with Graylog 4.3 the configuration is_master is replaced with is_leader. is_master still works but is deprecated.

The params.pp contains is_master and needs to be replaced.

metadata.json outputs module error with 'puppet module list'

Puppet-GrayLog 0.2.1 gives the following output when puppet module list is run:

$ puppet module list | egrep 'stdlib|apt'
Warning: Missing dependency 'puppetlabs-apt':
'graylog-graylog' (v0.2.1) requires 'puppetlabs-apt' (>= 2.2.2 < 3.0.0)
Warning: Missing dependency 'puppetlabs-stdlib':
'graylog-graylog' (v0.2.1) requires 'puppetlabs-stdlib' (>= 4.11.0 < 5.0.0)
âââ puppetlabs-apt (v2.3.0)
âââ puppetlabs-stdlib (v4.13.1)

This can be fixed by changing metadata.json:

"dependencies": [
- {"name":"puppetlabs-stdlib","version_requirement":">= 4.11.0 < 5.0.0"},
- {"name":"puppetlabs-apt","version_requirement":">= 2.2.2 < 3.0.0"}
+ {"name":"puppetlabs/stdlib","version_requirement":">= 4.11.0 < 5.0.0"},
+ {"name":"puppetlabs/apt","version_requirement":">= 2.2.2 < 3.0.0"}
],

(test on Ubuntu 16.04, Puppet 3.8.5)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.