robcowart / eslog_tutorial

From Raw Logs to Real Insights - A tutorial for getting started with log analytics using Elastic Stack.

From Raw Logs to Real Insights

I have a lot of passion for the Elastic Stack and the things it enables its users to achieve with their data. However the path to getting to this point was longer for me than it needed to be. With this tutorial material I am hoping to help make the same path shorter for others. So ...

if you want to know how to turn this...

<5>Oct 18 12:57:30 BRDC-2 kernel: [BlackRidge|Gateway|3.0.0.4619] class="Attribution" category="Unknown Identity" ctx="bump0" src="125.33.12.234" srcPort="25654" dest="5.149.112.53" destPort="23" identity="honeypot2Id" gwAction="DISCARD" gwMode="Monitor"

into this...

this tutorial is for you!

Back when I began my journey with the Elastic Stack I quickly discovered that while the online documentation provides a wealth of reference material, there was little that described what those first few steps should be. Online I found very little that covered more than the most basic tasks. Eventually as I stumbled upon more and more hints and tips, slowly things fell in place. Finally one day it really "clicked", and I have been enjoying the benefits of working with data in the Elastic Stack ever since. This tutorial follows very closely the exact path traveled as I took my first steps. I hope you find it helpful.

NOTE: The tutorial does not cover installing the Elastic Stack itself. This is covered well in the documentation. You will need Elasticsearch, Logstash and Kibana. Version 5.6.3 was used while creating this tutorial, but most versions should work fine.

The PDF file is the slide deck that I use to present the tutorial, although everything can also be presented using a live system. In fact the Kibana slides mostly provide examples, but by loading the dashboards from kibana/blackridge.kibana.json you can investigate how each visualization and dashboard was defined.

The files named logstash/##_blackridge.logstash.conf correspond to the step for building the pipeline in the slides. I encourage you to tryout each step for yourself and notice how each enhancement adds value to the raw data. The conf.d directory includes the final pipeline split into multiple files, demonstrating how you can keep the parts of a pipeline more managable by breaking it into multiple blocks of code.

The logs directory includes two files. dev.syslog contains a single syslog message that will be used as we develop the Logstash pipeline. data.syslog will be loaded once the pipeline is complete. It contains seven days of data, which we can analyze using Kibana dashboards.

If you find this material helpful or have any recommended improvements I would appreciate hearing your feedback.

Getting started with the Elastic Stack

If you are new to the Elastic Stack, this video goes beyond a simple default installation of Elasticsearch and Kibana. It discusses real-world best practices for hardware sizing and configuration, providing production-level performance and reliability.

Additionally local SSD storage should be considered as mandatory! For an in-depth look at how different storage options compare, and in particular how bad HDD-based storage is for Elasticsearch (even in multi-drive RAID0 configurations) you should watch this video...

eslog_tutorial's Issues

Odd error for pipeline

Rob, I have been following you for a while now. Absolutely love your work.

I have run into a somewhat odd issue I am hoping you can provide some input for. I am throwing up a new ES lab stack. CentOS 7/Logstash 6.2.3 alone on this box. I tried to do a very simple Syslog pipeline and got the below error. Confused (I have used this pipeline before) I decided to just throw your tutorial pipeline in unmodified. beyond ENV_VAR paths and got the exact same error. Your elastiflow pipeline works perfectly.....

Am I missing something stupidly obvious, or has something changed recently. This is my first 6.2x install.

Everything is in /etc/logstash/syslog/*

[user@logstash logstash]$ cat pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
# https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

• pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

• pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d"

• pipeline.id: syslog
  path.config: "/etc/logstash/syslog/conf.d"

[user@logstash logstash]$ sudo cat /etc/systemd/system/logstash.service.d/syslog.conf
#------------------------------------------------------------------------------
# Copyright (C)2018 KOIOSSIAN
#
# The contents of this file and/or repository are subject to the Koiossian
# Public License (the "License") and may not be used or distributed except in
# compliance with the License. You may obtain a copy of the License at:
#
# http://www.koiossian.com/public/koiossian_public_license.txt
#
# Software distributed under the License is distributed on an "AS IS" basis,
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for
# the specific language governing rights and limitations under the License.
#
# The Original Source Code was developed by Koiossian. Portions created by
# Koiossian are Copyright (C)2018 Koiossian. All Rights Reserved.
#------------------------------------------------------------------------------

[Service]
# Syslog
Environment="ESLOG_BASE=/etc/logstash/syslog/"
Environment="ESLOG_DICTIONARY_PATH=$ESLOG_BASE/dictionaries"
Environment="ESLOG_GROK_PATTERNS_DIR=$ESLOG_BASE/patterns"
Environment="ESLOG_GEOIP_DBS_DIR=$ESLOG_BASE/geoipdbs"
Environment="ESLOG_TEMPLATE_PATH=$ESLOG_BASE/templates"
Environment="ESLOG_LOGS=$ESLOG_BASE/logs"
Environment="ESLOG_ELASTICSEARCH_HOSTS=elastic:9200"
Environment="ESLOG_ELASTICSEARCH_USER=elastic"
Environment="ESLOG_ELASTICSEARCH_PASSWORD=changeme"
Environment="ESLOG_SYSLOG_PORT=514"
Environment="ESLOG_SYSLOG_TZ=UTC"

[logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:syslog, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:89:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:128:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:87:in plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:112:in plugin'", "(eval):255:in '", "org/jruby/RubyKernel.java:994:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:84:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}