doytsujin / lumberjack

A simple, powerful, and very fast logging utility that can be a drop in replacement for Logger or ActiveSupport::BufferedLogger. Provides support for automatically rolling log files even with multiple processes writing the same log file.

Lumberjack

Lumberjack is a simple, powerful, and fast logging implementation in Ruby. It uses nearly the same API as the Logger class in the Ruby standard library and as ActiveSupport::BufferedLogger in Rails.

Usage

This code aims to be extremely simple to use. The core interface it the Lumberjack::Logger which is used to log messages (which can be any object) with a specified Severity. Each logger has a level associated with it and messages are only written if their severity is greater than or equal to the level.

  logger = Lumberjack::Logger.new("logs/application.log")  # Open a new log file with INFO level
  logger.info("Begin request")
  logger.debug(request.params)  # Message not written unless the level is set to DEBUG
  begin
    # do something
  rescue => exception
    logger.error(exception)
    raise
  end
  logger.info("End request")

This is all you need to know to log messages.

Features

Meta data

When messages are added to the log, additional data about the message is kept in a Lumberjack::LogEntry. This means you don't need to worry about adding the time or process id to your log messages as they will be automatically recorded.

The following information is recorded for each message:

• severity - The severity recorded for the message.
• time - The time at which the message was recorded.
• program name - The name of the program logging the message. This can be either set for all messages or customized with each message.
• process id - The process id (pid) of the process that logged the message.
• unit of work id - The unique 12 byte hexadecimal number generated for a unit of work.

Units Of Work

A unit of work can be used to tie together all log messages within a block. This can be very useful to isolate a group of messages that represent one path through the system. For instance, in a web application, a single request represents a natural unit of work, and when you are looking through a log file, it is useful to see the entire set of log entries as a unit instead of interspersed with messages from other concurrent requests.

  # All log entries in this block will get a common unit of work id.
  Lumberjack.unit_of_work do
    logger.info("Begin request")
    yield
    logger.info("End request")
  end

Pluggable Devices

When a Logger logs a LogEntry, it sends it to a Lumberjack::Device. Lumberjack comes with a variety of devices for logging to IO streams or files.

• Lumberjack::Device::Writer - Writes log entries to an IO stream.
• Lumberjack::Device::LogFile - Writes log entries to a file.
• Lumberjack::Device::DateRollingLogFile - Writes log entries to a file that will automatically roll itself based on date.
• Lumberjack::Device::SizeRollingLogFile - Writes log entries to a file that will automatically roll itself based on size.
• Lumberjack::Device::Null - This device produces no output and is intended for testing environments.

If you'd like to send you log to a different kind of output, you just need to extend the Device class and implement the +write+ method. Or check out these plugins:

• lumberjack_syslog_device - send your log messages to the system wide syslog service
• lumberjack_multi-device - send log messages to multiple devices
• lumberjack_mongo_device - store your log messages to a MongoDB NoSQL data store
• lumberjack-couchdb-driver - store your log messages to a CouchDB NoSQL data store
• lumberjack_heroku_device - log to Heroku's logging system

Customize Formatting

When a message is logged, it is first converted into a string. You can customize how it is converted by adding mappings to a Formatter.

  logger.formatter.add(Hash, :pretty_print)  # use the Formatter::PrettyPrintFormatter for all Hashes
  logger.formatter.add(MyClass){|obj| "#{obj.class}@#{obj.id}"}  # use a block to provide a custom format

If you use the built in devices, you can also customize the Template used to format the LogEntry.

  # Change the format of the time in the log
  Lumberjack::Logger.new("application.log", :time_format => "%m/%d/%Y %H:%M:%S")

  # Use a simple template that only includes the time and the message
  Lumberjack::Logger.new("application.log", :template => ":time - :message")

  # Use a custom template as a block that only includes the first character of the severity
  template = lambda{|e| "#{e.severity_label[0, 1]} #{e.time} - #{e.message}"}
  Lumberjack::Logger.new("application.log", :template => template)

Buffered Performance

The logger has hooks for devices that support buffering to increase performance by batching physical writes. Log entries are not guaranteed to be written until the Lumberjack::Logger#flush method is called.

You can use the :flush_seconds option on the logger to periodically flush the log. This is usually a good idea so you can more easily debug hung processes. Without periodic flushing, a process that hangs may never write anything to the log because the messages are sitting in a buffer. By turning on periodic flushing, the logged messages will be written which can greatly aid in debugging the problem.

The built in stream based logging devices use an internal buffer. The size of the buffer (in bytes) can be set with the :buffer_size options when initializing a logger. The default behavior is to not to buffer.

  # Set buffer to flush after 8K has been written to the log.
  logger = Lumberjack::Logger.new("application.log", :buffer_size => 8192)
  
  # Turn off buffering so entries are immediately written to disk.
  logger = Lumberjack::Logger.new("application.log", :buffer_size => 0)

Automatic Log Rolling

The built in devices include two that can automatically roll log files based either on date or on file size. When a log file is rolled, it will be renamed with a suffix and a new file will be created to receive new log entries. This can keep your log files from growing to unusable sizes and removes the need to schedule an external process to roll the files.

There is a similar feature in the standard library Logger class, but the implementation here is safe to use with multiple processes writing to the same log file.

Examples

These example are for Rails applications, but there is no dependency on Rails for using this gem. Most of the examples are applicable to any Ruby application.

In a Rails application you can replace the default production logger by adding this to your config/environments/production.rb file:

  # Use the ActionDispatch request id as the unit of work id. This will use just the first chunk of the request id.
  # If you want to use an abbreviated request id for terseness, change the last argument to `true`
  config.middleware.insert_after ActionDispatch::RequestId, Lumberjack::Rack::RequestId, false
  # Use a custom unit of work id to each request
  # config.middleware.insert(0, Lumberjack::Rack::UnitOfWork)
  # Change the logger to use Lumberjack
  log_file_path = Rails.root + "log" + "#{Rails.env}.log"
  config.logger = Lumberjack::Logger.new(log_file, :level => :warn)

To set up a logger to roll every day at midnight, you could use this code (you can also specify :weekly or :monthly):

  config.logger = Lumberjack::Logger.new(log_file_path, :roll => :daily)

To set up a logger to roll log files when they get to 100Mb, you could use this:

  config.logger = Lumberjack::Logger.new(log_file_path, :max_size => 100.megabytes)

To change the log message format, you could use this code:

  config.logger = Lumberjack::Logger.new(log_file_path, :template => ":time - :message")

To change the log message format to output JSON, you could use this code (this example requires the multi-json gem):

  config.logger = Lumberjack::Logger.new(log_file_path, :template => lambda{|e| MultiJson.dump(e)})

To send log messages to syslog instead of to a file, you could use this (require the lumberjack_syslog_device gem):

  config.logger = Lumberjack::Logger.new(Lumberjack::SyslogDevice.new)