y-matsuwitter / embulk-parser-fluent-s3-log

Embulk parser plugin for fluent-plugin-s3's output file.

Fluent S3 Log parser plugin for Embulk

This plugin parses fluent-plugin-s3's output log file.

Overview

• Plugin type: parser
• Guess supported: no

Configuration

• columns: column name and its type. (array, required)

Example

in:
  type: any file input plugin type
  parser:
    type: fluent-s3-log
    columns:
      - {name: id, type: long}
      - {name: path, type: string}

Build

$ rake

embulk-parser-fluent-s3-log's Issues

Use Java::LineDecoder

Ver0.0.1 reads all lines per each log files to parse.
This causes large memory consumption if a log file is huge.
In order to avoid this, it's recommended to use Java::LineDecoder or something else to use buffers efficiently.

See: https://gist.github.com/frsyuki/9c4128cd888cf12551b8