Add beats output routing about elk-docker

[spujadas]

Sorry, not quite sure what the exact issue is here, it really should work out-of-the-box. 😕

The output filter/routing for Logstash is in 30-output.conf, which contains a minimal configuration item:

output {
  elasticsearch { hosts => ["localhost"] }
  stdout { codec => rubydebug }
}

So implicitly, in the elasticsearch section, the default values are used for non-specified configuration options (as per https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html), e.g. "logstash-%{+YYYY.MM.dd}" for index.

Using the vanilla sebp/elk image on a clean VM and having Filebeat push logs from an instance of nginx to Logstash's Beat input plugin on port 5044 (see the example in the documentation of the image) produces an entry like this when browsing to a page served by nginx:

{
  "_index": "logstash-2016.01.07",
  "_type": "nginx-access",
  "_id": "AVIdlGrkng6MqhVcdOZ3",
  "_score": null,
  "_source": {
    "message": "XX.XX.XX.XX - - [07/Jan/2016:19:33:19 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\" \"-\"",
    "@version": "1",
    "@timestamp": "2016-01-07T19:33:23.023Z",
    "beat": {
      "hostname": "ac29184dfcf0",
      "name": "ac29184dfcf0"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "offset": 0,
    "source": "/var/log/nginx/access.log",
    "type": "nginx-access",
    "host": "ac29184dfcf0",
    "clientip": "XX.XX.XX.XX",
    "ident": "-",
    "auth": "-",
    "timestamp": "07/Jan/2016:19:33:19 +0000",
    "verb": "GET",
    "request": "/",
    "httpversion": "1.1",
    "response": "304",
    "bytes": "0",
    "agent": "\"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\""
  },
  "fields": {
    "@timestamp": [
      1452195203023
    ]
  },
  "sort": [
    1452195203023
  ]
}

… which looks fine to me.

Using the piece of configuration you suggested (which I added to 30-output.conf where it would belong), the same operation creates entries such as this:

{
  "_index": "filebeat-2016.01.07",
  "_type": "nginx-access",
  "_id": "AVIdkPy-3efy8HwjSlbN",
  "_score": null,
  "_source": {
    "message": "XX.XX.XX.XX - - [07/Jan/2016:19:29:31 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\" \"-\"",
    "@version": "1",
    "@timestamp": "2016-01-07T19:29:40.787Z",
    "beat": {
      "hostname": "86fe311709cb",
      "name": "86fe311709cb"
    },
    "count": 1,
    "fields": null,
    "input_type": "log",
    "offset": 0,
    "source": "/var/log/nginx/access.log",
    "type": "nginx-access",
    "host": "86fe311709cb",
    "clientip": "XX.XX.XX.XX",
    "ident": "-",
    "auth": "-",
    "timestamp": "07/Jan/2016:19:29:31 +0000",
    "verb": "GET",
    "request": "/",
    "httpversion": "1.1",
    "response": "304",
    "bytes": "0",
    "agent": "\"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0\""
  },
  "fields": {
    "@timestamp": [
      1452194980787
    ]
  },
  "sort": [
    1452194980787
  ]
}

… so essentially the same thing, except for the _index field which has a different prefix (default logstash vs explicitly set filebeat).

Having said that, there may be something wrong that I'm not seeing or I may be misunderstanding the issue: if so could you please provide steps to reproduce the issue you're having? Cheers.

[sra]

The issue is that if you configure the beat to send directly to elastic search, the information ends up in filebeat-XXXX.XX.XX or topbeat-XXXX.XX.XX indexes. If you send through the logstash beat plugin as configured, the information ends up in the logstash-* index. It seems like it should be the same in either case.

Additionally, the beats project provides a bunch of pre-made dashboards which only work with the information in the XXXbeat-XXXX.XX.XX format.

https://www.elastic.co/guide/en/beats/libbeat/current/getting-started.html#load-kibana-dashboards

[spujadas]

Right, got it this time. I thought the issue was about the plugin not working rather than about inconsistent behaviours and predefined dashboards not playing properly.
Will update in a sec.