swillis11 / logsandra

A Cassandra demo application, log management

Logsandra

Logsandra is a log management application written in Python that uses Cassandra for storage.

The project is open source under the MIT license. See LICENSE for more info.

Requirements

Logsandra works with Cassandra 0.7 or 0.8.

The following Python packages are required:

* Pylons >= 1.0
* Jinja2
* PyYAML
* pycassa >= 0.5.4
* python-dateutil
* CherryPy >= 3.1
* Thrift >= 0.5.0
* Cassandra (not automatically installed during the installation phase)

Installation

To install Logsandra run:

pip install -r requirements.txt
python setup.py install

All Python modules will be automatically installed by that command except for the Cassandra bindings.

To generate the keyspace and column family needed by logsandra, run scripts/create_keyspace.py.

Configuration

The main configuration file for Logsandra is logsandra.yaml. Below is a list of configuration variables (including type and optional/required):

ident               identity of the logsandra server [String, Optional/Discovery]

httpd_host          host on which to serve the logsandra web service [String, Required]
httpd_port          port on which to serve the logsandra web service [String, Required]
httpd_config        name of the pylons configuration file [String]

cassandra_host      cassandra host to connect too [String, Required]
cassandra_port      cassandra port to connect too [String, Required]
cassandra_timeout   in seconds until connection is aborted if no response is delivered [String, Optional]

paths               list of paths to monitor [List, Optional]
  path              directory or file [String, Required]
  recursive         recursively enter sub directories [True/False, Optional]
  parser            dictionary containing parser name and arguments [Dictionary, Required]
    name            name of the parser [String, Required]
    ...             arguments, see specific parser [Depends, Optional]

Logsandra is built on Pylons and therefore a Pylons configuration file should be present for Logsandra's web service to work.

Usage

Logsandra consists of two main scripts, both of which act as daemons.

Usage:

python logsandra-httpd.py [options] start|stop|restart
python logsandra-monitord.py [options] start|stop|restart

If you want a list of options invoke either one these commands:

python logsandra-httpd.py --help
python logsandra-monitord.py --help

A typical setup is to run logsandra-httpd on one server and logsandra-monitord on all servers producing log files. logsandra-monitord store the logs in the Cassandra cluster.

Create Your Own Parser

You can create your own Logsandra parser by adding a module to the logsandra.monitor.parsers package.

An example parser:

from logsandra.monitor.parsers import BaseParser

class FooParser(BaseParser):
    def parse(self, line, source, data):
        date = None
        keywords = []
        return self.client.add_log(date=date, entry=line, source=source, keywords=keywords)

You should also look at logsandra/monitor/parsers/ for more parser examples.