y-ken / fluent-plugin-twitter

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.

fluent-plugin-twitter

Component

Fluentd Input/Output plugin to process tweets with Twitter Streaming API.

Dependency

before use, install dependent library as:

# for RHEL/CentOS (eventmachine requires build dependency)
$ sudo yum -y install gcc gcc-c++ openssl-devel libcurl libcurl-devel

# for Ubuntu/Debian (eventmachine requires build dependency)
$ sudo apt-get install build-essential libssl-dev

Requirements

fluent-plugin-twitter	fluentd	ruby
0.6.1	v0.14.x	>= 2.1
0.5.4	v0.12.x	>= 1.9

Installation

install with gem or td-agent-gem command as:

# for fluentd
$ gem install eventmachine
$ gem install fluent-plugin-twitter

# for td-agent2
$ sudo td-agent-gem install eventmachine
$ sudo td-agent-gem install fluent-plugin-twitter -v 0.5.4

Input Configuration

Input Sample

It require td-agent2 (fluentd v0.12) to use keyword with hashtag.

<source>
  @type twitter
  consumer_key        YOUR_CONSUMER_KEY        # Required
  consumer_secret     YOUR_CONSUMER_SECRET     # Required
  access_token        YOUR_ACCESS_TOKEN        # Required
  access_token_secret YOUR_ACCESS_TOKEN_SECRET # Required
  tag                 input.twitter.sampling   # Required
  timeline            tracking                 # Required (tracking or sampling or location or userstream)
  keyword             'Ruby,Python,#fleuntd'   # Optional (keyword has priority than follow_ids)
  follow_ids          14252,53235              # Optional (integers, not screen names)
  locations           31.110283, 129.431631, 45.619283, 145.510175  # Optional (bounding boxes; first pair specifies longitude/latitude of southwest corner)
  lang                ja,en                    # Optional
  output_format       nest                     # Optional (nest or flat or simple[default])
  flatten_separator   _                        # Optional
</source>

<match input.twitter.sampling>
  @type stdout
</match>

Proxy support

<source>
  @type twitter
  consumer_key        YOUR_CONSUMER_KEY        # Required
  consumer_secret     YOUR_CONSUMER_SECRET     # Required
  access_token        YOUR_ACCESS_TOKEN        # Required
  access_token_secret YOUR_ACCESS_TOKEN_SECRET # Required
  tag                 input.twitter.sampling   # Required
  timeline            tracking                 # Required (tracking or sampling or location or userstream)
  <proxy>
    host proxy.example.com                     # Required
    port 8080                                  # Required
    username proxyuser                         # Optional
    password proxypass                         # Optional
  </proxy>
</source>

Testing

$ tail -f /var/log/td-agent/td-agent.log

Output Configuration

Output Sample

<source>
  @type http
  port 8888
</source>

<match notify.twitter>
  @type twitter
  consumer_key        YOUR_CONSUMER_KEY
  consumer_secret     YOUR_CONSUMER_SECRET
  access_token        YOUR_ACCESS_TOKEN
  access_token_secret YOUR_ACCESS_TOKEN_SECRET
</match>

Proxy support

<match notify.twitter>
  @type twitter
  consumer_key        YOUR_CONSUMER_KEY
  consumer_secret     YOUR_CONSUMER_SECRET
  access_token        YOUR_ACCESS_TOKEN
  access_token_secret YOUR_ACCESS_TOKEN_SECRET
  <proxy>
    host proxy.example.com
    port 8080
    username proxyuser
    password proxypass
  </proxy>
</match>

Testing

$ curl http://localhost:8888/notify.twitter -F 'json={"message":"foo"}'

Reference

Twitter OAuth Guide

http://pocketstudio.jp/log3/2012/02/12/how_to_get_twitter_apikey_and_token/

Quick Tour to count a tweet with fluent-plugin-twitter and fluent-plugin-datacounter

http://qiita.com/items/fe4258b394190f23fece

TODO

patches welcome!

Copyright

Copyright © 2012- Kentaro Yoshida (@yoshi_ken)

License

Apache License, Version 2.0

fluent-plugin-twitter's Issues

プラグインの設定を入れると、「process finished code=6」で落ちます

お忙しいところ失礼します。
m-masakiと申します。

CentOS6.4にてtd-agentをインストールして、fluentdを使用しています。
通常は起動できるのですが、

このプラグインを試してみようと思い、以下の設定を入れたところ、起動した瞬間にプロセスが落ちてしまいまして、ちょっと困っています。

以下の<source>~</source>をコメントアウトしますと、普通に起動できます。
td-agent.logの内容も以下に書いておりますが、「process finished code=6」と出てますが、調べたのですが、よくわかりません...

もしご存知であれば、何か情報等はありませんでしょうか？
こちらの情報で不足ありましたら、別途書きますので、よろしくお願いします。

■td-agent.confの内容

type twitter consumer_key ・・・・・・・・・・ consumer_secret ・・・・・・・・・・ oauth_token ・・・・・・・・・・ oauth_token_secret ・・・・・・・・・・ tag debug.twitter timeline userstream lang ja,en output_format flat

■td-agent.logの内容
2013-05-02 18:50:30 +0900: twitter: starting Twitter Streaming API for userstream. tag:debug.twitter lang:ja,en
2013-05-02 18:50:30 +0900: process finished code=6
2013-05-02 18:50:30 +0900: process died within 1 second. exit.

undefined method `emit' for nil:NilClass

Hi @okkez

I'm afraid do you know the solve this problem?

repeatable example

startup with following command. and exit wide ctrl-c.

$ bundle exec fluentd --conf fluentd_twitter.conf

error output

2017-04-03 11:47:57 +0900 [info]: #0 shutting down input plugin type=:twitter plugin_id="object:3fedc78fd44c"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down input plugin type=:twitter plugin_id="object:3fedc78f50e4"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:twitter plugin_id="object:3fedc78e7d68"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3fedc6013abc"
2017-04-03 11:47:57 +0900 [info]: #0 shutting down output plugin type=:stdout plugin_id="object:3fedc6c97bbc"
2017-04-03 11:47:57 +0900 [warn]: #0 thread exited by unexpected error plugin=Fluent::Plugin::TwitterInput title=:in_twitter error_class=NoMethodError error="undefined method `emit' for nil:NilClass"
2017-04-03 11:47:57 +0900 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `emit' for nil:NilClass"

keyword "カレー"

I'm having some serious issues with fluentd-0.12.29 + gem 'fluent-plugin-twitter' version '0.5.4'.
If I set keyword parameter like

keyword 'カレー' # Optional (keyword has priority than follow_ids)

then giving me the following error on startup,

Starting td-agent: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/config/basic_parser.rb:56:in `skip': invalid byte sequence in US-ASCII (ArgumentError)

Is there any way to use this special words for me ?
Thanks.

Gem conflict error: json-2.0.2 conflicts with json (~> 1.8)

I am trying to set this up on Docker using 'fluentd-0.12.31' Fluentd image and it seems the plugin requires implicitly json-1.8 while fluent requires json-2.0. Is there any hard requirement for json-1.8 and can we update the dependencies?

2017-02-28 18:39:01 +0000 [info]: adding source type="twitter"
/usr/lib/ruby/2.3.0/rubygems/specification.rb:2284:in `raise_if_conflicts': Unable to activate twitter-5.17.0, because json-2.0.2 conflicts with json (~> 1.8) (Gem::ConflictError)
        from /usr/lib/ruby/2.3.0/rubygems/specification.rb:1407:in `activate'
        from /usr/lib/ruby/2.3.0/rubygems.rb:196:in `rescue in try_activate'
        from /usr/lib/ruby/2.3.0/rubygems.rb:193:in `try_activate'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:125:in `rescue in require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:40:in `require'
        from /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-twitter-0.6.0/lib/fluent/plugin/in_twitter.rb:1:in `<top (r
equired)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `each'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:170:in `try_load_plugin'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:130:in `new_impl'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/plugin.rb:55:in `new_input'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:149:in `add_source'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:95:in `block in configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `each'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/root_agent.rb:92:in `configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:129:in `configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/engine.rb:103:in `run_configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:489:in `run_configure'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:160:in `block in start'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:366:in `main_process'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `fork'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:338:in `supervise'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/supervisor.rb:156:in `start'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.31/bin/fluentd:5:in `<top (required)>'
        from /usr/bin/fluentd:23:in `load'
        from /usr/bin/fluentd:23:in `<main>'

log_level enabled can not do?

Fill the console you get lots of Tweets, are in trouble.

Add log_level option by repeatedly · Pull Request #255 · fluent/fluentd

Plug-in stopped working

We've been using this plug-in successfully for months, but it just recently stopped working with no changes on our end. It starts up and listens for tweets but does not receive any, nor does it receive any error messages. Is it possible the latest upgrade to Twitters API to 1.1 broke this plug-in? If so, an upgrade would be much appreciated. Thank you!

introduce proxy support

https://github.com/sferik/twitter/blob/master/examples/Configuration.md#using-a-proxy

new function (connection via proxy)

Can you please build a proxy config in the module? I need to be able to configure a proxy to acces the outside world.

Does it support UTF-8?

Hi,
I'm putting non-english string on keyword. However it gives me :
2016-01-08 04:09:21 -0500 [info]: twitter: starting Twitter Streaming API for tracking. tag:twitter keyword:[utf-8characters]
but doesn't working.

conflict http_parser.rb

Dear y-ken san

I'm inokappa.

I tried td-agent and fluent-plugin-twitter.
I got an error message(please confirm following "message and backtrace") while trying to "/etc/init.d/td-agent configtest".

message and backtrace

• https://gist.github.com/inokappa/7727216

my environment

• td-agent_1.1.17
• fluent-plugin-twitter 0.2.2

and other gems.

/usr/lib/fluent/ruby/bin/fluent-gem list | grep tw
em-twitter (0.3.1)
fluent-plugin-twitter (0.2.2)
tweetstream (2.6.0)
twitter (4.8.1)

td-agent.conf

<source>
  type twitter
  consumer_key        xxxxxxxxxxxxxxxxxxxxxx
  consumer_secret     yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  oauth_token         zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz                      
  oauth_token_secret  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  tag                 input.twitter.sampling  
  timeline            sampling
  keyword             hoge
  lang                ja,en
</source>

<match input.twitter.sampling>
  type stdout
</match>

Any advice or information you could provide would be greatly appreciated.

Thank you.

No output from Twitter stream

I'm not sure is this a miss-configuration or something more, but I have to output from Twitter stream. Configs are:

• Ubuntu 14.04
• Ruby 2.1.2
• fluentd and fluent-plugin-twitter installed as gems

fluentd.config:

<source>
  type twitter
  consumer_key ...API key from app.twitter.com...
  consumer_secret ...API secret from app.twitter.com...
  oauth_token ...Access token from app.twitter.com...
  oauth_token_secret ... Access token secret from app.twitter.com...
  tag                 input.twitter
  timeline            sampling
</source>

<match input.twitter>
  type stdout
</match>

Then I start fluentd as fluentd -c fluentd.conf -o ./out.log -v

Here is the output:

</ROOT>
2014-06-09 05:33:04 -0400 [info]: fluent/engine.rb:101:block in configure: adding source type="twitter"
2014-06-09 05:33:05 -0400 [info]: fluent/engine.rb:117:block in configure: adding match pattern="input.twitter" type="stdout"
2014-06-09 05:33:05 -0400 [info]: plugin/in_twitter.rb:74:get_twitter_connection: twitter: starting Twitter Streaming API for sampling. tag:input.twitter keyword:ruby,rails,job
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:272:block in install_supervisor_signal_handlers: fluentd supervisor process get SIGINT
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:379:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2014-06-09 05:33:53 -0400 [debug]: fluent/supervisor.rb:382:block in install_main_process_signal_handlers: getting start to shutdown main process
2014-06-09 05:33:53 -0400 [info]: fluent/engine.rb:230:run: shutting down fluentd
2014-06-09 05:33:53 -0400 [info]: fluent/supervisor.rb:236:supervise: process finished code=0
2014-06-09 05:35:47 -0400 [info]: fluent/supervisor.rb:219:supervise: starting fluentd-0.10.49
2014-06-09 05:35:47 -0400 [info]: fluent/supervisor.rb:321:read_config: reading config file path="fluentd.conf"
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-datacounter' version '0.4.3'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-elasticsearch' version '0.3.0'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-mongo' version '0.7.3'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-td' version '0.10.20'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-twitter' version '0.3.0'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:87:block in configure: gem 'fluentd' version '0.10.49'
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:91:configure: using configuration file: <ROOT>
  <source>
    type twitter
    consumer_key .......
    consumer_secret ...........
    oauth_token ..........
    oauth_token_secret ........
    tag input.twitter
    timeline sampling
    keyword ruby,rails,job
  </source>
  <match input.twitter>
    type stdout
  </match>
</ROOT>
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:101:block in configure: adding source type="twitter"
2014-06-09 05:35:47 -0400 [info]: fluent/engine.rb:117:block in configure: adding match pattern="input.twitter" type="stdout"
2014-06-09 05:35:47 -0400 [info]: plugin/in_twitter.rb:74:get_twitter_connection: twitter: starting Twitter Streaming API for sampling. tag:input.twitter keyword:ruby,rails,job
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:272:block in install_supervisor_signal_handlers: fluentd supervisor process get SIGINT
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:379:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2014-06-09 05:36:04 -0400 [debug]: fluent/supervisor.rb:382:block in install_main_process_signal_handlers: getting start to shutdown main process
........

I suppose here I have to see tweets stream, but nothing appears.

Something is wrong with output too.

Here is the config:

<source>
  type http
  port 8888
</source>

<match out.twitter>
  consumer_key ...
  consumer_secret ...
  oauth_token ...
  oauth_token_secret ...
  type twitter
</match>

Then I do:

curl http://localhost:8888/out.twitter -F 'json={"message":"hello from fluentd-twitter"}'

and output is:

2014-06-09 06:31:23 -0400 [warn]: fluent/engine.rb:156:rescue in emit_stream: emit transaction failed  error_class=NoMethodError error=#<NoMethodError: undefined method `update' for #<Twitter::Client:0x000000029a32b8>

Did I used Twitter's API/Access token and secrets correctly (Twitter's terminology differs, however I tried to swap API/ACCESS with no results - no errors, no output)?

Any ideas what is wrong?

how would i compile a new gem plugin to install such that I can have more fields from twitter api?

how would i compile a new gem plugin to install such that I can have more fields from twitter api?

Also wondering, why the whole tweet object was not kept intact

Gem::ConflictError: Unable to activate activesupport-5.2.3, because tzinfo-2.0.0 conflicts with tzinfo (~> 1.1)

Hi folks,

I am using td-agent 3.5.1 and fluent-plugin-sql 1.1.1. When I start the service then a gem conflict error occurs:

-- A start job for unit td-agent.service has begun execution.
-- 
-- The job identifier is 4773.
  fluentd[8590]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:2290:in `raise_if_conflicts': Unable to activate activesupport-5.2.3, because tzinfo-2.0.0 conflicts with tzinfo (~> 1.1) (Gem::ConflictError)
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1410:in `activate'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1444:in `block in activate_dependencies'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1430:in `each'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1430:in `activate_dependencies'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:1412:in `activate'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems.rb:220:in `rescue in try_activate'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems.rb:213:in `try_activate'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:126:in `rescue in require'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:39:in `require'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/in_sql.rb:23:in `<module:Plugin>'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/in_sql.rb:21:in `<top (required)>'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/registry.rb:102:in `block in search'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/registry.rb:99:in `each'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/registry.rb:99:in `search'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/registry.rb:44:in `lookup'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/plugin.rb:146:in `new_impl'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/plugin.rb:100:in `new_input'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:315:in `add_source'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:160:in `block in configure'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:156:in `each'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/root_agent.rb:156:in `configure'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/engine.rb:131:in `configure'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/engine.rb:96:in `run_configure'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:812:in `run_configure'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:589:in `dry_run'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:607:in `supervise'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/supervisor.rb:512:in `run_supervisor'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/lib/fluent/command/fluentd.rb:324:in `<top (required)>'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  fluentd[8590]:         from /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.7.4/bin/fluentd:8:in `<top (required)>'
  fluentd[8590]:         from /opt/td-agent/embedded/bin/fluentd:23:in `load'
  fluentd[8590]:         from /opt/td-agent/embedded/bin/fluentd:23:in `<main>'
  systemd[1]: td-agent.service: Control process exited, code=exited, status=1/FAILURE

and my td-agent.conf is below:

# system settings
<system>
  log_level warn
  workers 8
</system>

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  @type forward
  @id input_forward
  tag app.log
</source>

# sp logs
<worker 0>
 <source>
  @type sql
  @log_level warn

  host localhost
  database postgres_demo
  adapter postgresql
  username ***
  password ***
 
  tag_prefix local.postgres  

  select_interval 5s  
  select_limit 500     

  state_file /var/run/fluentd/sql_state

  <table>
   table public.sample
   tag demo.optional
   update_column id
  </table>

 </source>
</worker>

## live debugging agent
<source>
  @type debug_agent
  @id input_debug_agent
  bind 127.0.0.1
  port 24230
</source>

# parse JSON log field
<filter app.log>
 @type parser
 key_name log
 reserve_time true
 reserve_data true
 remove_key_name_field true
 <parse>
   @type json
 </parse>
</filter>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
  @id output_stdout
</match>


## elastic search output
<match app.log local.postgres**>
  @type elasticsearch
  host localhost
  port 9200
  index fluentd
  logstash_format true
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  logstash_prefix fluentd
  <buffer>
    @type file
    path /var/log/td-agent/buffer/td
  </buffer>
</match>

tzinfo 2.0 conflictis with tzinfo 1.1, how could I fix this? Is there an update for this problem?

Thank you.

Release new version for avoiding twitterstream gem problems.

The master uses twitter gem instead of twitterstream.
Release new version, maybe 0.6?, is better for users.
twitterstream is unmaintained gem and it has a problem with ruby 2.4.