ksurent / es-utils

ElasticSearch Utilities

NAME

App::ElasticSearch::Utilities - Utilities for Monitoring ElasticSearch

VERSION

version 3.3

SYNOPSIS

This library contains utilities for unified interfaces in the scripts.

This a set of utilities to make monitoring ElasticSearch clusters much simpler.

Included are:

SEARCHING:

scripts/es-search.pl - Utility to interact with LogStash style indices from the CLI

MONITORING:

scripts/es-nagios-check.pl - Monitor ES remotely or via NRPE with this script
scripts/es-graphite-dynamic.pl - Perform index maintenance on daily indexes
scripts/es-status.pl - Command line utility for ES Metrics
scripts/es-storage-data.pl - View how shards/data is aligned on your cluster
scripts/es-nodes.pl - View node information

MAINTENANCE:

scripts/es-daily-index-maintenance.pl - Perform index maintenance on daily indexes
scripts/es-alias-manager.pl - Manage index aliases automatically
scripts/es-open.pl - Open any closed indices matching a index parameters

MANAGEMENT:

scripts/es-copy-index.pl - Copy an index from one cluster to another
scripts/es-apply-settings.pl - Apply settings to all indexes matching a pattern
scripts/es-storage-data.pl - View how shards/data is aligned on your cluster

DEPRECATED:

scripts/es-graphite-static.pl - Send ES Metrics to Graphite or Cacti

The App::ElasticSearch::Utilities module simply serves as a wrapper around the scripts for packaging and distribution.

FUNCTIONS

es_pattern

Returns a hashref of the pattern filter used to get the indexes { string => '*', re => '.*', }

es_connect

Without options, this connects to the server defined in the args. If passed an array ref, it will use that as the connection definition.

es_request([$handle],$command,{ method => 'GET', parameters => { a => 1 } }, {})

Retrieve URL from ElasticSearch, returns a hash reference

First hash ref contains options, including:

uri_param           Query String Parameters
index               Index name
type                Index type
method              Default is GET

es_nodes

Returns the hash of index meta data.

es_indices_meta

Returns the hash of index meta data.

es_indices

Returns a list of active indexes matching the filter criteria specified on the command line. Can handle indices named:

logstash-YYYY.MM.DD
dcid-logstash-YYYY.MM.DD
logstash-dcid-YYYY.MM.DD
logstash-YYYY.MM.DD-dcid

Makes use of --datesep to determine where the date is.

Options include:

• state

  Default is 'open', can be used to find 'closed' indexes as well.

• check_state

  Default is 1, set to 0 to disable state checks. The combination of the default with this option and the default for state means only open indices are returned.

• check_dates

  Default is 1, set to 0 to disable checking index age.

es_index_days_old( 'index-name' )

Return the number of days old this index is.

es_index_shard_replicas( 'index-name' )

Returns the number of replicas for a given index.

es_index_valid( 'index-name' )

Checks if the specified index is valid

es_index_fields('index-name')

Returns a list of the fields in a given index.

es_close_index('index-name')

Closes an index

es_open_index('index-name')

Open an index

es_delete_index('index-name')

Deletes an index

es_optimize_index('index-name')

Optimize an index to a single segment per shard

es_index_segments( 'index-name' )

Exposes GET /$index/_segments

Returns the segment data from the index in hashref:

es_segment_stats($index)

Return the number of shards and segments in an index as a hashref

es_index_stats( 'index-name' )

Exposes GET /$index/_stats

Returns a hashref

es_settings()

Exposes GET /_settings

Returns a hashref

es_node_stats()

Exposes GET /_nodes/stats

Returns a hashref

def('key')

Exposes Definitions grabbed by options parsing

ARGS

From App::ElasticSearch::Utilities:

--local         Use localhost as the elasticsearch host
--host          ElasticSearch host to connect to
--port          HTTP port for your cluster
--noop          Any operations other than GET are disabled
--timeout       Timeout to ElasticSearch, default 30
--keep-proxy    Do not remove any proxy settings from %ENV
--index         Index to run commands against
--base          For daily indexes, reference only those starting with "logstash"
                 (same as --pattern logstash-* or logstash-DATE)
--datesep       Date separator, default '.' also (--date-separator)
--pattern       Use a pattern to operate on the indexes
--days          If using a pattern or base, how many days back to go, default: all

ARGUMENT GLOBALS

Some options may be specified in the /etc/es-utils.yaml or $HOME/.es-utils.yaml file:

---
host: esproxy.example.com
port: 80
timeout: 10

INSTALL

This library attempts to provide scripts compatible with version 0.19 through 1.1 of ElasticSearch.

Recommended install with CPAN Minus:

cpanm App::ElasticSearch::Utilities

You can also use CPAN:

cpan App::ElasticSearch::Utilities

Or if you'd prefer to manually install:

export RELEASE=<CurrentRelease>

wget --no-check-certificate https://github.com/reyjrar/es-utils/blob/master/releases/App-ElasticSearch-Utilities-$RELEASE.tar.gz?raw=true -O es-utils.tgz

tar -zxvf es-utils.tgz

cd App-ElasticSearch-Utilities-$RELEASE

perl Makefile.PL

make

make install

This will take care of ensuring all the dependencies are satisfied and will install the scripts into the same directory as your Perl executable.

USAGE

The tools are all wrapped in their own documentation, please see:

$UTILITY --help
$UTILITY --manual

For individual options and capabilities

PATTERNS

Patterns are used to match an index to the aliases it should have. A few symbols are expanded into regular expressions. Those patterns are:

*       expands to match any number of any characters.
?       expands to match any single character.
DATE    expands to match YYYY.MM.DD, YYYY-MM-DD, or YYYYMMDD
ANY     expands to match any number of any characters.

CONTRIBUTORS

Mihai Oprea <[email protected]>
Samit Badle

AUTHOR

Brad Lhotsky [email protected]

COPYRIGHT AND LICENSE

This software is Copyright (c) 2012 by Brad Lhotsky.

This is free software, licensed under:

The (three-clause) BSD License