fluent-plugin-elasticsearch, geoip, typecast Together and slow。 about fluent-plugin-elasticsearch

[zbyufei]

new config with buffer plugin like this:

<match es.typecast.access>
  type elasticsearch
  host 127.0.0.1
  port 9200
  logstash_prefix logstash
  logstash_format true
  type_name fluentd
  index_name fluentd
  flush_interval 5s

  buffer_type memory
  flush_interval 60
  retry_limit 17
  retry_wait 1.0
  num_threads 2
</match>

I am a beginners for fluent and elasticsearch. I want to cat logfile(>10G) to clickstream-access.log but so slowly.

[pitr]

Not sure what geoip and typecast plugins do, but we have 'flush_interval 30s' in our configs for elasticsearch plugin. Try decreasing that value. Also, try playing with buffer_chunk_limit and friends (see http://docs.fluentd.org/articles/buffer-plugin-overview)

I guess you've already seen #3?

[zbyufei]

yes. I have read #3 .
Thanks, I tried to remove geoip and typecast to see how the performance.

[zbyufei]

I've done the test and found to be elasticsearch index performance problems.
I was reading through elasticsearch issue and elasticsearch google groups to improve its performance.
Here's a relevant performance test reports:

http://blog.sematext.com/2013/07/08/elasticsearch-refresh-interval-vs-indexing-performance/ :

refresh_interval: 1s   – 2.0K docs/s
refresh_interval: 5s   – 2.5K docs/s
refresh_interval: 30s – 3.4K docs/s

Thanks and I close this issue.