kazegusuri / fluent-plugin-bufferize

A fluentd plugin that enhances existing non-buffered output plugin as buffered plugin.

fluent-plugin-bufferize, a plugin for Fluentd

An adapter plugin which enables existing non-buffered plugins to resend messages easily in case of unexpected exceptions without creating duplicate messages.

Why

Buffered plugin accumulates many messages in buffer and sends all the messages at a same time. There are many APIs that does not support such reqeusts like bulk-insert. In that case, you have to use non-buffered output and implement resend meschanism yourself because non-buffered output lacks exception handling functionality.

To use this plugin, you just have to create non-buffered plugin without caring exception handling. If an exception happens in your plugin, the request is issued again automatically. With file buffer, none of messages are lost even on sudden fluentd process down.

Configuration

Just embrace existing configuration by directive.

If you use following configuration:

<match *>
  type http
  endpoint_url    http://foo.bar.com/
  http_method     put
</match>

Modify it like this:

<match *>
  type bufferize
  buffer_type file
  buffer_path /var/log/fluent/myapp.*.buffer
  <config>
    type http
    endpoint_url    http://foo.bar.com/
    http_method     put
  </config>
</match>

This is a buffered output plugin. For more information about parameters, please refer official document.

Example of application

These plugins are good compatibility to fluent-plugin-bufferize.

• fluent-plugin-out-http
• fluent-plugin-jubatus
• fluent-plugin-irc

Installation

Add this line to your application's Gemfile:

gem 'fluent-plugin-bufferize'

And then execute:

$ bundle

Or install it yourself as:

$ gem install fluent-plugin-bufferize

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Author	Masahiro Sano
Copyright	Copyright (c) 2013- Masahiro Sano
License	MIT License

fluent-plugin-bufferize's Issues

Keep original timestmap

Hi,
I am using this plugin in combination with https://github.com/ento/fluent-plugin-out-http to send logs from supervisor (which users this script: https://github.com/Globegitter/supervisor2fluentd/blob/nanosecond-precision/supervisor2fluentd.py). This python script emits an event to fluent with tag and timestamp which I then want to forward on. However I have notice that all the logs coming in with the same batch have the same timestamp (well, different on subsecond level), even though in reality they are logged out at different times. If I reduce the flush_interval from e.g. 10s to 1s I get the accurate timestamps.

Is this intended with this plugin and is there some way to preserve the original timestamp I am emitting?

I am using buffer_type file btw, not sure if it makes any difference,