fluent / fluentd-docs Goto Github PK

View Code? Open in Web Editor NEW

49.0 18.0 119.0 19.13 MB

This repository is deprecated. Go to fluentd-docs-gitbook repository.

Ruby 45.31% CSS 38.17% JavaScript 9.26% Shell 0.45% HTML 6.81%

fluentd document

fluentd-docs's Introduction

Fluentd Docs App

Important Warnings

THE OFFICIAL DOCUMENTATION FOR FLUENTD HAS BEEN MOVED TO GITBOOK

This repository is kept as a historical archive. If you find something wrong on https://docs.fluentd.org, please submit issues or PRs to the new GitBook repository.

https://github.com/fluent/fluentd-docs-gitbook

Overview

Ultrasimple CMS and content for Fluentd documentation. The production site is here.

If you'd like to propose an edit to the Fluentd docs, please fork this repo and send us a pull request.

Note that we require each commit to be signed off by the author as policy. Thus, when writing a patch, please sign your commit using the -s option.

# Append `--amend` to sign the previous commit
$ git commit -s

Install

$ gem install bundler
$ bundle install --path vendor/bundle
$ bundle exec rake server
$ open "http://localhost:9395/"

Test

$ bundle exec rake test

Build Search Index

$ heroku run rake index

Deploy

We deploy fluentd-docs by Circle-CI automatically. If you want to deploy it manually, run following command.

$ git push heroku master

NOTE

When you have updated an article, please update config/last_updated.json too.

$ bundle exec rake last_updated
$ git add config/last_updated.json

Contributing docs for v1

v1 docs are under docs/v1. Currently, most articles are symlinked to the corresponding file in docs/.

If you are updating an exiting article. Just replace the symlink with an actual article.
If you are adding a new article, just add it under docs/v1.

Once v1 is released, this process will be updated.

INCLUDE pragma

You can use the "INCLUDE pragma" to avoid copy-and-pasting the same content or updates on multiple pages.

The syntax is as follows:

INCLUDE: <filename without extension>

... the rest of the document.

Please remember to include a blank line between "INCLUDE..." and the rest of the document. The docs app will search for <filename>.txt in the docs directory and insert its contents into the current document.

For example, if you write

INCLUDE: _buffer_parameters

... the rest of the document.

then the docs app will insert the contents of _buffer_parameters.txt into the current document.

Acknowledgement

This program is forked from heroku/heroku-docs, and originally written by @rtomayko and @adamwiggins. Later, modified by @kzk and @doryokujin.

Code is released under the MIT License: http://www.opensource.org/licenses/mit-license.php

fluentd-docs's People

Contributors

Stargazers

Watchers

Forkers

ijin oza tagomoris chocoby komamitsu jmoses choplin elia cesarandreu willnet jney pombredanne tatsuyafw sonots y-ken rwstauner yubobo spsusbilla kevinchow hoshi-sano akira-kuriyama orfah mcaramello methane mahito jderrett mr-salty image-tester hernandesbsousa sho-h ctohastek alex-ec2 schmooster hkmurakami takumakanari udzura yoshi-taka viljaste ajohnstone inhwankim syohex graingert uu59 adamchainz occho documentations toyama0919 nikushi namusyaka mparienti hfm voxik tbenade jungsuhan muojp dwoo4dwoo yoheimuta okkez cosmo0920 ota42y wuub milk knqyf263 csharpru fissible nokute78 rcoh johnlinp kenhys toru-takahashi jcaxmacher agup006 whirlwin mururu k0kubun yurano ryysud salewski shillasaebi kozdincer rtusys akerouanton tmshn ritwik28 niroyb mr-brody hayamiz optionalg eagletmt martinssipenko francjohny yaron-idan pbialasiewicz j-un fri-day captn3m0 jmcabandara dukex algestam runningstream

fluentd-docs's Issues

Various issues with http://docs.fluentd.org/articles/in_forward

Hi team,

Thanks for the nice documentation.

However, I found some problems with http://docs.fluentd.org/articles/in_forward

re: This plugin uses MessagePack for the internal protocol.

Actually it accepts JSON too.

echo -n '["debug.access", 1308466941, {"a":1}]' | telnet localhost 24224

works fine.. this shows up in the log

2011-06-19 07:02:21 +0000 debug.access: {"a":1}

as does multiple values:

 echo -n '["debug.access", 1308466941, {"a":1}]["debug.access", 1308466942, {"b":2}]' | telnet localhost 24224

and the alternate format

echo -n '["debug.access", [[1308466941, {"a":1}], [1308466942, {"b":2}]]]' | telnet localhost 24224

It would be nice to have the exact field and type formats documented (Instead of having to decode binary data streams).
Explicitly say what the timestamp is... it appears to be a standard Unix-seconds-since-Epoch. Using unsigned int 32 in messagepack?
I think the 'message' is a string, but a fully formed JSON object. ' {"a":1, ... } '

re: examples

This example is incorrect?

  ["myapp.access", [1308466941, {"a"=>1}], [1308466942, {"b"=>2}]]

missing '[' in front of '[1308466941'
it's missing a final ']'
also since JSON is actually supported, perhaps using JSON ':' instead of '=>' might make sense.

Hmm, I just realised this is on github, so I can fork and make a pull request?

thanks all!

nickg

Update "Free Alternative to Splunk Using Fluentd" document with Kibana v3

Many people now use Kibana v3

There are not in_unix article link in Input Plugins menu of Japanese language

I expects that there are in_unix article link in Input Plugins menu of Japanese language but are not.

Missing GPLv2 license notice

I'm trying to find the GPLv2 license notice that should be visible and mentioned in the text at http://docs.fluentd.org/articles/cep-norikra.

Download Docs as PDF

I have provided my email in the pop-up box and waited for few days.
It's simply not working. I've tried few times and I also checked my junk/spam box .
This is my first time using fluentd

Thank you for your support !

http://docs.fluentd.org/articles/quickstart

Typo in docs

Documentation: http://docs.fluentd.org/articles/architecture
Heading: Reliable Buffering

"A new empty chunk is pushed onto the top of the queue when either-

The size of the top chunk reaches its limit, or-
The timer expires."

Isn't a not missing?

Document the timezone field once it is released.

fluent/fluentd#430

https://github.com/fluent/fluentd-docs/blob/master/docs/_formatter_common_parameters.txt needs to be updated.

Add the filter article for v1 docs

Filters are introduced in v0.12.

td-agent doesnt deploy on debian 7 with libssl1.0.0

I got this error when i try to deploy the td-agent.deb

dpkg --install td-agent_1.1.9-1_amd64.deb
(Reading database ... 56569 files and directories currently installed.)
Preparing to replace td-agent 1.1.9-1 (using td-agent_1.1.9-1_amd64.deb) ...
Unpacking replacement td-agent ...
dpkg: dependency problems prevent configuration of td-agent:
td-agent depends on libssl0.9.8 (>= 0.9.8k-1); however:
Package libssl0.9.8 is not installed.

dpkg: error processing td-agent (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
td-agent

But, the current stable version of libssl is on the system :

apt-cache policy libssl1.0.0
libssl1.0.0:
Installed: 1.0.1e-2
Candidate: 1.0.1e-2
Version table:
*** 1.0.1e-2 0
500 http://ftp.belnet.be/debian/ wheezy/main amd64 Packages
100 /var/lib/dpkg/status

Quid?

Document plugin dir for td-agent

T/O

Quickstart.txt has broken link to plugin.

< a href="/plugin/"> is broken link.
Is correct link http://fluentd.org/plugin/?

Recipe/syslog/elasticsearch - add in "logstash_format true"

I found this special variable in documentation on the http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd page.

<source>
  type syslog
  port 42185
  tag syslog
</source>

<source>
  type forward
</source>

<match syslog.**>
  type elasticsearch
  logstash_format true
  flush_interval 10s # for testing
</match>

It would help to also have this information on the elasticsearch recipe pages as well since people will also be looking there for the Kibana / ElasticSearch solution. This is what is shown now:

<source>
  type syslog
  port 5140
  bind 0.0.0.0
  tag system.local
</source>

<match **>
  type elasticsearch
  host <hostname> #(optional; default="localhost")
  port <port> #(optional; default=9200)
  index_name <index name> #(optional; default=fluentd)
  type_name <type name> #(optional; default=fluentd)
</match>

When using Kibana, the data isn't sent to ElasticSearch in the right format unless logstash_format is set to true.

Thanks

Doc for out_file talks about mongo

On http://docs.fluentd.org/articles/out_file the following paragraph exists:

num_threads

The number of threads to flush the buffer. This option can be used to 
parallelize the data insert into MongoDB. The default is 1.

I'm guess this is an error. If it is not, there needs to be more discussion about why out_file has a parameter related to mongo.

Add plugin use-case pages

Currently, Fluentd has many plugins and combinations.
So we need the page which explains useful plugin combination for several situations.

Add each Output explanation

Documentation describes Output classes on only plugin dev page.
In this result, user sometimes confused, "This is a TimeSlicedOutput plugin. Why this plugin doesn't flush the chunk?"

We should add each Output explanation on Output page and each plugin links to corresponding Output class explanation.

Break down the plugin development page

Currently, docs.fluentd.org/v0.12/articles/plugin-development has too much content. It should be broken down to separate pages per plugin (output, input, filter, parser, formatter) with more detailed guidelines on each page.

In particular, each page should have the following:

When should I develop my custom input/output/filter/parser/formatter plugin?
Have you already looked at ?
How to write a custom plugin.
How to release a custom plugin.

Then, docs.fluentd.org/v0.12/articles/plugin-development should really become just an overview page with links to the individual pages.

in_multiprocess article doesn't have ja link in 'Available languages'

in_multiprocess article doesn't have ja link in 'Available languages'. It is expected that it has ja link in 'Available languages'.
http://docs.fluentd.org/articles/in_multiprocess

Replace Overview presentation slide with new slide.

This slide has several outdated content.
We should update this slide for introduction, architecture overview and comparison with clear message.

td-agent does not install cleanly with Homebrew on Mavericks

I get this on Homebrew:

jace@razor ~ $ brew install "http://toolbelt.treasuredata.com/brew/td-agent.rb"
######################################################################## 100.0%
==> Cloning https://github.com/treasure-data/td-agent.git
Updating /Library/Caches/Homebrew/td-agent--git
==> Checking out revision 34a4dfc56fe44b3c14e7b1b234a178feaaeab705
==> RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/Cellar/openssl/1.0.1e --w
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install bundler -v 1.3.5 --no-ri -
ERROR:  Could not find a valid gem 'bundler' (= 1.3.5) in any repository
ERROR:  Possible alternatives: bundler

READ THIS: https://github.com/Homebrew/homebrew/wiki/troubleshooting

Installation proceeds if I install bundler manually, but brew can't symlink td-agent into /usr/local after this:

[1] jace@razor ~ $ gem install bundler -v 1.3.5                      [15:55:53]
zsh: correct 'bundler' to '.bundler' [nyae]? n
Successfully installed bundler-1.3.5
Parsing documentation for bundler-1.3.5
1 gem installed
jace@razor ~ $ brew install "http://toolbelt.treasuredata.com/brew/td-agent.rb"
######################################################################## 100.0%
==> Cloning https://github.com/treasure-data/td-agent.git
Updating /Library/Caches/Homebrew/td-agent--git
==> Checking out revision 34a4dfc56fe44b3c14e7b1b234a178feaaeab705
==> RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/Cellar/openssl/1.0.1e --w
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install bundler -v 1.3.5 --no-ri -
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install msgpack -v 0.4.7 --no-ri -
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install iobuffer -v 1.1.2 --no-ri 
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install cool.io -v 1.1.1 --no-ri -
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install http_parser.rb -v 0.5.1 --
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install yajl-ruby -v 1.1.0 --no-ri
==> Fluentd revision: f7105bc435abd5f4e74f13c5e05c625808ba1912
==> git clone https://github.com/fluent/fluentd.git
==> git checkout f7105bc435abd5f4e74f13c5e05c625808ba1912
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem build fluentd.gemspec
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install ./fluentd-*.gem --no-ri --
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install td-client -v 0.8.55 --no-r
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install td -v 0.10.89 --no-ri --no
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-td -v 0.10.1
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install thrift -v 0.8.0 --no-ri --
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-scribe -v 0.
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-flume -v 0.1
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install bson -v 1.8.6 --no-ri --no
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install bson_ext -v 1.8.6 --no-ri 
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install mongo -v 1.8.6 --no-ri --n
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-mongo -v 0.7
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install nokogiri -v 1.5.10 --no-ri
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install aws-sdk -v 1.8.3.1 --no-ri
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-s3 -v 0.3.4 
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install webhdfs -v 0.5.3 --no-ri -
==> /usr/local/Cellar/td-agent/1.1.17/bin/gem install fluent-plugin-webhdfs -v 0
==> Caveats
td-agent configuration file and plugin directories were created:

    /usr/local/etc/td-agent/td-agent.conf
    /usr/local/etc/td-agent/plugin

You can invoke td-agent manually via td-agent command without launchctl:

    td-agent --pid /usr/local/var/run/td-agent/td-agent.pid

If you want to know the details of Fluentd, see Fluentd documents at:

    http://docs.fluentd.org/

To have launchd start td-agent at login:
    ln -sfv /usr/local/opt/td-agent/*.plist ~/Library/LaunchAgents
Then to load td-agent now:
    launchctl load ~/Library/LaunchAgents/homebrew.mxcl.td-agent.plist
Warning: Could not link td-agent. Unlinking...
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
You can try again using `brew link td-agent'

Possible conflicting files are:
/usr/local/bin/testrb -> /usr/local/Cellar/ruby/2.0.0-p247/bin/testrb
/usr/local/bin/ruby -> /usr/local/Cellar/ruby/2.0.0-p247/bin/ruby
/usr/local/bin/ri -> /usr/local/Cellar/ruby/2.0.0-p247/bin/ri
/usr/local/bin/rdoc -> /usr/local/Cellar/ruby/2.0.0-p247/bin/rdoc
/usr/local/bin/rake -> /usr/local/Cellar/ruby/2.0.0-p247/bin/rake
/usr/local/bin/irb -> /usr/local/Cellar/ruby/2.0.0-p247/bin/irb
/usr/local/bin/gem -> /usr/local/Cellar/ruby/2.0.0-p247/bin/gem
/usr/local/bin/erb -> /usr/local/Cellar/ruby/2.0.0-p247/bin/erb
/usr/local/include/yaml.h -> /usr/local/Cellar/libyaml/0.1.4/include/yaml.h
/usr/local/share/man/man1/ruby.1 -> /usr/local/Cellar/ruby/2.0.0-p247/share/man/man1/ruby.1
/usr/local/share/man/man1/ri.1 -> /usr/local/Cellar/ruby/2.0.0-p247/share/man/man1/ri.1
/usr/local/share/man/man1/rake.1 -> /usr/local/Cellar/ruby/2.0.0-p247/share/man/man1/rake.1
/usr/local/share/man/man1/irb.1 -> /usr/local/Cellar/ruby/2.0.0-p247/share/man/man1/irb.1
/usr/local/share/man/man1/erb.1 -> /usr/local/Cellar/ruby/2.0.0-p247/share/man/man1/erb.1
/usr/local/lib/pkgconfig/yaml-0.1.pc -> /usr/local/Cellar/libyaml/0.1.4/lib/pkgconfig/yaml-0.1.pc
/usr/local/lib/libyaml.dylib -> /usr/local/Cellar/libyaml/0.1.4/lib/libyaml.dylib
/usr/local/lib/libyaml.a -> /usr/local/Cellar/libyaml/0.1.4/lib/libyaml.a
/usr/local/lib/libyaml-0.2.dylib -> /usr/local/Cellar/libyaml/0.1.4/lib/libyaml-0.2.dylib
==> Summary
X  /usr/local/Cellar/td-agent/1.1.17: 17138 files, 126M, built in 31.3 minutes

I'm not a Ruby developer. I only depend on a few Ruby gems such as Compass and Sass. What should I do here?

Add "by EsperTech" to "Esper" please

Please add the text "by EsperTech" to all usages of the name "Esper" please, for fair use of Esper and the EsperTech trademark. Please link to "http://esper.codehaus.org" or "http://www.espertech.com" for any links that pertain to Esper (instead of linking directly to the doc page).

Write the "life of a Fluentd event" page

In v0.12, the event routing mechanism has changed for Fluentd. It's a good opportunity to create a page that gives an overview of how events come into Fluentd, how it is routed, etc.

fluent/supervisor.rb:205:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'elasticsearch'. Run 'gem search -rd fluent-plugin' to find plugins"

Even if I do what this error suggests I still get it when trying to start td-agent on CentOS 6.5.

I've installed the gem with gem install fluent-plugin-elasticsearch and I've also tried with fluent-gem install fluent-plugin-elasticsearch.

Any idea what I'm doing wrong?

Can't install on Raspberry Pi

~ $ fluentd -c fluent.conf
/usr/lib/ruby/1.9.1/rubygems/dependency.rb:247:in to_specs': Could not find fluentd (>= 0) amongst [addressable-2.2.8, af-0.3.19, bundler-1.7.2, bundler-unload-1.0.2, executable-hooks-1.3.2, gem-wrappers-1.2.5, interact-0.4.8, json_pure-1.6.8, mime-types-2.3, netrc-0.7.7, rb-readline-0.4.2, rest-client-1.7.0.rc1, rubygems-bundler-1.4.4, rubyzip-0.9.9, rvm-1.11.3.9, terminal-table-1.4.5, uuidtools-2.1.5] (Gem::LoadError) from /usr/lib/ruby/1.9.1/rubygems/dependency.rb:256:into_spec'
from /usr/lib/ruby/1.9.1/rubygems.rb:1231:in gem' from /usr/local/bin/fluentd:22:in

~ $ curl -X POST -d 'json={"sensor1":3123.13,"sensor2":321.3}' \

http://localhost:8888/td.testdb.raspberrypi
curl: (7) couldn't connect to host

~ $ rvm current
ruby-2.1.2

Difference between fluentd and td-agent

Paragraph "What's td-agent?" says:

That’s why Treasure Data, Inc is providing the stable distribution of Fluentd, called td-agent. The differences between Fluentd and td-agent can be found here.

and links to FAQ that doesn't have that question. It should probably link to this.

Document secondary option

Seems that this option is not documented in the new documentation website.

Thanks,
Sylvain

Add mixin page

For users (configuration) and developers (include usage)

Issue installing td-agent in os x maverics

While doing: brew install "http://toolbelt.treasuredata.com/brew/td-agent.rb"
Getting errors like:

...
Then to load td-agent now:
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.td-agent.plist
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/readline.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/openssl.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/openssl.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/sha2.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/sha2.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/sha1.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/sha1.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/rmd160.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/rmd160.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/md5.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/md5.bundle malformed object (unknown load command 3)
/usr/bin/install_name_tool: object: /usr/local/Cellar/td-agent/1.1.17/lib/libyaml-0.2.dylib malformed object (unknown load command 4)

If I start td-agent:

2014-01-25 13:17:32 +0530 [info]: starting fluentd-0.10.39
2014-01-25 13:17:32 +0530 [info]: reading config file path="/usr/local/etc/td-agent/td-agent.conf"
/usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require': dlopen(/usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/md5.bundle, 9): Library not loaded: @@HOMEBREW_PREFIX@@/opt/openssl/lib/libcrypto.1.0.0.dylib (LoadError) Referenced from: /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/md5.bundle Reason: image not found - /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/x86_64-darwin13.0.2/digest/md5.bundle from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/webrick/httpauth/digestauth.rb:17:in <top (required)>' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/webrick/httpauth.rb:12:in<top (required)>'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/webrick.rb:227:in <top (required)>' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin/in_monitor_agent.rb:22:inclass:MonitorAgentInput'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin/in_monitor_agent.rb:19:in <module:Fluent>' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin/in_monitor_agent.rb:18:in<top (required)>'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin.rb:60:in block in load_plugin_dir' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin.rb:58:ineach'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin.rb:58:in load_plugin_dir' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/plugin.rb:52:inload_plugins'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/engine.rb:46:in init' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:319:ininit_engine'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:88:in block in start' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:196:incall'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:196:in main_process' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:171:inblock in supervise'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:170:in fork' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:170:insupervise'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/supervisor.rb:85:in start' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/lib/fluent/command/fluentd.rb:141:in<top (required)>'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in require' from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/1.9.1/rubygems/custom_require.rb:55:inrequire'
from /usr/local/Cellar/td-agent/1.1.17/lib/ruby/gems/1.9.1/gems/fluentd-0.10.39/bin/fluentd:6:in <top (required)>' from /usr/local/Cellar/td-agent/1.1.17/bin/fluentd:23:inload'
from /usr/local/Cellar/td-agent/1.1.17/bin/fluentd:23:in <top (required)>' from /usr/local/bin/td-agent:5:inload'
from /usr/local/bin/td-agent:5:in `

'
2014-01-25 13:17:32 +0530 [info]: process finished code=256
2014-01-25 13:17:32 +0530 [warn]: process died within 1 second. exit.

My brew config:
dmins-MacBook-Pro:~ admin$ brew --config
HOMEBREW_VERSION: 0.9.5
ORIGIN: https://github.com/Homebrew/homebrew
HEAD: 0e7e7544441184a11231c7dd70f6b742e5299a13
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
CPU: quad-core 64-bit haswell
OS X: 10.9.1-x86_64
CLT: 5.0.1.0.1.1382131676
Clang: 5.0 build 500
X11: N/A
System Ruby: 1.8.7-358
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby

Missing ltsv and nginx format template doc

There are no description about ltsv and nginx format template at http://docs.fluentd.org/articles/in_tail

Monitoring / Alerting for web services article

Many users use Fluentd for web service monitoring using several plugins.
Adding this article is helpful for new Fluentd users.

@sonots wrote following articles in Japanese.
We can write the article based on these posts.

http://blog.livedoor.jp/sonots/archives/25189820.html
http://blog.livedoor.jp/sonots/archives/25018617.html

Recommend json or msgpack in out_exec_filter

Currently, out_exec_filter uses TSV by default.
But TSV seems rare in some use-cases.
If we have a chance, then we should change the default format to json (I like msgpack).

If we can't change the default format, we should make supporting format more obvious in the docs. Some user says "Why out_exec_filter supports only TSV?"

Add the label article for v1 docs

Labels are added in v0.12

brew install "http://toolbelt.treasuredata.com/brew/td-agent.rb" is not working

fluentd installation instruction for mac is not accurate.
http://toolbelt.treasuredata.com/brew/td-agent.rb :: the URL is outdated (returns 404)

nginx error log format is wrong

I read this (http://docs.fluentd.org/articles/common-log-formats) and set nginx error log format

<source>
    type tail
    format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>[^#]*)#(?<tid>[^:]*): (?<message>.*)$/
    tag nginx.error
    path /var/log/nginx/error.log
</source>

But I met configtest error:

# service td-agent configtest
2014-08-13 10:23:15 -0400 [error]: Dry run failed: Unknown format template '/^(?<time>[^ ]+ [^ ]+) [(?<log_level>.*)] (?<pid>[^'

I think # acts comment out, and this works fine:

<source>
    type tail
    format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>[^\#]*)\#(?<tid>[^:]*): (?<message>.*)$/
    tag nginx.error
    path /var/log/nginx/error.log
</source>

Now /var/log/td-agent/td-agent.log shows:

<source>
    type tail
    format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>[^#]*)#(?<tid>[^:]*): (?<message>.*)$/
    tag nginx.error
    path /var/log/nginx/error.log
</source>

My environment:

# td-agent --version
td-agent 0.10.52
# dpkg -l|grep td-agent
ii  td-agent                            2.0.4-0                              amd64        Treasure Agent: A data collector for Treasure Data
# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"
# uname -a
Linux dev-do.mouse.tachikoma.io 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Add example script to in_exec, out_exec and out_exec_filter

It helps new users.

Double html escape for tilte

http://docs.fluentd.org/articles/signals

<title>Fluentd&amp;rsquo;s Signal Handling | Fluentd</title>

should be corrected as

<title>Fluentd's Signal Handling | Fluentd</title>

Spelling mistake in docs "High-Availablability Configurations of Fluentd"

At http://docs.fluentd.org/articles/perl
noticed a spelling mistake with the heading:
"High-Availablability Configurations of Fluentd"

Presume it should be:
"High-Availability Configurations of Fluentd"

Line 84, https://github.com/fluent/fluentd-docs/blob/master/docs/perl.txt, for example

Create an "internals" for v0.12

Since v0.12 will have two big changes in Fluentd's internals (labels and filters), let's make a page that describes how Fluentd works at the internals level.

Add td-agent article

Currently, we describe the differences on both fluentd.org and docs.fluentd.org.
I think separating contents is bit messy and not user friendly.

Fluentd newbies often ask td-agent questions on mailing list, so
we should improve td-agent related contents.
I think one page article which collects td-agent information is enough.

Content images:

What is td-agent
td-agent settings, e.g. configuration file, log file and etc.
How to manage plugin on td-agent
The differences between td-agent and td-agent2
etc...

I will do PR later.

memory Buffer Plugin buffer_queue_limit should be 64

memory Buffer Plugin's buffer_queue_limit was changed from 256 to 64 by fluent/fluentd@f70fa37#diff-1f30c01be5408d714bda4d75efd8e17b

Please sync the doc at http://docs.fluentd.org/articles/buf_memory with the source code.

Pattern not match on AWS EC-2

I was install td-agent on EC 2, i using configure following:

<source>
  type tail
  path /var/log/nginx/access.log
  pos_file /var/log/td-agent/td.nginx.access.pos
  format nginx
  tag td.nginx.access
</source>

<match td.nginx.access>
    type s3
    aws_key_id XXXXXXXXX
    aws_sec_key XXXXXXXXXXXXXXXXXXXXXXX
    s3_bucket [My bucket, don not ask me, please]
    buffer_path /var/log/td-agent/s3
    time_slice_format %Y%m%d%H%M%S
    flush_interval 1m
    check_apikey_on_start false
    buffer_chunk_limit 256m
</match>

It not work, and give my log:

2014-04-24 09:02:14 +0000 [warn]: pattern not match: "118.69.63.140 - - [24/Apr/2014:09:02:14 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36\" \"-\""
2014-04-24 09:02:14 +0000 [warn]: pattern not match: "118.69.63.140 - - [24/Apr/2014:09:02:14 +0000] \"GET /poweredby.png HTTP/1.1\" 404 3696 \"http://54.255.129.91/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36\" \"-\""

Can you help me fix this log, i was trying but it not done!

What's the intent of the out_exec_filter? What does it do with the records it reads back from an external program stdout?

Seems to me http://docs.fluentd.org/articles/out_exec_filter is missing something critical here. I can succesfully process the records in an external program, but what does Fluentd do with the records that get returned on the external program stdout? What's the actual intent of the out_exec_filter plugin?

Document that files found by Gem.find_files('fluent_plugin') are loaded

It wasn't obvious to me that files of the path format lib/fluent/plugin/<TYPE>_<NAME>.rb are loaded if and only if the type/name combination is specified somewhere in the configuration.

It would be helpful to have something along the line (*) of

If your plugin is not going to be directly referenced from the config, name your plugin file like fluent_plugin.<EXT> (EXT being .rb, .so, etc) and place it at the root of your gem’s require_path. Those files are discovered via Gem::find_files then loaded. Take care when implementing a plugin this way as your plugin file may be loaded multiple times if multiple versions of your gem are installed.

in the Installing custom plugins section of the docs.

(*) Copied and modified from the Rubygems doc.

"Parsing Common Log Formats" Recipe doesn't exist

In http://docs.fluentd.org/categories/recipes, when you click on "Parsing Common Log Formats", you go to http://docs.fluentd.org/articles/common-log-formats which show "Internal server error"

Add the description about num_threads

RethinkDB output plugin

Can we mention this plugin to output log to RethinkDB? https://github.com/kureikain/fluent-plugin-rethink

It's very similar(but less features and less mature) to MongoDB plugin.

Clean up categories

I noticed some articles are placed under incorrect category.

trouble-shooting, signals and etc articles are in Configuration.
Community is in Developer.
etc...

We should move above artciles into correct category.

In out_forward documentation, incorrect default value was described.

In http://docs.fluentd.org/ja/articles/out_forward#numthreads:
"書き込みのリトライ間隔とリトライ数。デフォルト値は各々、1.0と1.7です。" is incorrect.
It may be "書き込みのリトライ間隔とリトライ数。デフォルト値は各々、1.0と17です。"
thank you.

Red hat rpm install fails, URL not found.

http://docs.fluentd.org/articles/install-by-rpm

When running the install script, it attempts to access this URL that fails:

http://packages.treasuredata.com/2/redhat/latest/x86_64/repodata/repomd.xml

Document various types supported in Fluentd

Let's document all the types supported by config.rb

Add tag routing illustration to Config File page

Current 'Config File' article is based on Text.
It hards to image Fluentd Input / Output relation and tag handling.
Adding image helps understanding how tag routing works.