Json fields parser and Logstash_prefix problem about fluent-bit-kubernetes-logging

[karol-brejna-i]

Any news on this one?

[edsiper]

Sorry for the delay, for some reason I did not see this one.

What about if you remove:

      Index fluentbit
      Type flb_type

[karol-brejna-i]

We had config without Index and Type (we tried to rely on default values).
When LogstashFormat was on, the the index was created with a name like -2017.05.01 - without any prefix.

This is "smaller" issue for us, though. Not serializing json object as expected (as we had it working in fluentd) is more "serious".

Please, advise what to check/fix.

[edsiper]

I've logged this issue to troubleshoot and fix it:

fluent/fluent-bit#256

[edsiper]

@karol-brejna-i @darek-dymek-i,

In Fluent Bit 0.11 I have done the following improvements:

• Fix missing logstash prefix
• Add new merge_json_log option to filter_kubernetes

These changes are not officially released and I need your help to test them to get some feedback. Steps:

• Pull new Fluent Bit image from Docker Hub (alter your daemonset file) fluent/fluent-bit:0.11.5-test
• In your ConfigMaps, append the option "merge_json_log on" in the Kubernetes filter section
• Deploy the new daemon set

If is OK, I will go ahead and push the new release.

[dardymek]

Thanks @edsiper but unfortunately it doesn't create any index in elastic
My conf in configmap:

    [SERVICE]
      Flush        1
      Daemon       Off
      Log_Level    debug
      Parsers_File /fluent-bit/etc/parsers.conf

    [INPUT]
      Name          tail
      Tag           kubernetes.*
      Path          /var/log/containers/*.log
      Parser        docker
      DB            /var/log/flb_kube.db
      Mem_Buf_Limit 5MB

    [FILTER]
      Name     kubernetes
      Match    kubernetes.*
      Kube_URL https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
      merge_json_log on

    [OUTPUT]
      Name  es
      Match *
      Host  ${FLUENT_ELASTICSEARCH_HOST}
      Port  ${FLUENT_ELASTICSEARCH_PORT}
      Index fluentbit
      Logstash_Format Off
      Logstash_Prefix logtest
      Retry_Limit False

I tried for:

• Logstash_Format On with default Logstash_Prefix
• Logstash_Format On with Logstash_Prefix = logtest
• Logstash_Format Off with Index = fluentbit
  in all cases there was no any index in es

logs from fluentbit:

[2017/05/11 10:01:22] [ info] [engine] started
[2017/05/11 10:01:22] [debug] [in_tail] inotify watch fd=20
[2017/05/11 10:01:22] [debug] [in_tail] scanning path /var/log/containers/*.log
[2017/05/11 10:01:22] [debug] [in_tail] add to scan queue /var/log/containers/bash-3250055436-4g66k_default_bash-02d0970b0ae595034c71291f241a0702e7944c609e75b13e675d794f85baf869.log, offset=329
.....
[2017/05/11 10:01:22] [debug] [in_tail] add to scan queue /var/log/containers/ubuntu-3236687841-ckjlm_default_ubuntu-3bd6ed13b6102a70852f20697d1f3b3877039b2e3ffae11b86eca77095500e5e.log, offset=593
[2017/05/11 10:01:22] [debug] [out_es] host=elasticsearch-logging port=9200 index=fluentbit type=flb_type
[2017/05/11 10:01:22] [ info] [filter_kube] https=1 host=10.3.0.1 port=443
[2017/05/11 10:01:22] [ info] [filter_kube] local POD info OK
[2017/05/11 10:01:22] [ info] [filter_kube] testing connectivity with API server...
[2017/05/11 10:01:22] [debug] [filter_kube] API Server (ns=kube-system, pod=fluent-bit-es-t12d9) http_do=0, HTTP Status: 200
[2017/05/11 10:01:22] [ info] [filter_kube] API server connectivity OK
[2017/05/11 10:01:22] [debug] [router] input=tail.0 'DYNAMIC TAG'
[2017/05/11 10:01:22] [debug] [in_tail] file=/var/log/containers/bash-3250055436-4g66k_default_bash-02d0970b0ae595034c71291f241a0702e7944c609e75b13e675d794f85baf869.log promote to TAIL_EVENT
[2017/05/11 10:01:22] [debug] [in_tail] file=/var/log/containers/bash-3250055436-4g66k_default_bash-aff39e3baf36d01b13059610f0e68d321af002649be3413d106523feef2c767e.log promote to TAIL_EVENT
[2017/05/11 10:01:22] [debug] [in_tail] file=/var/log/containers/ccc-access-control-claim-data-62340211-2xvs9_default_ccc-access-control-claim-data-5713fa1dd84dc50eb9a1c52396cf901e5400e8de6021b71080eaf0b50236cc01.log promote to TAIL_EVENT
[2017/05/11 10:01:22] [debug] [in_tail] file=/var/log/containers/ccc-access-control-claim-data-62340211-2xvs9_default_ccc-access-control-claim-data-e1f949dc9a63a414690cf6769f394e4e5f9f6fa645fbf055d22dbf28c945235f.log promote to TAIL_EVENT
[2017/05/11 10:01:22] [debug] [filter_kube] API Server (ns=default, pod=ccc-access-control-claim-data-frontend-3507161469-z5s1w) http_do=0, HTTP Status: 200

[edsiper]

Thanks for your feedback.

I've pushed some improvements that likely address the issue you are facing (I was able to reproduce the same behavior here).

Would you please re-pull a fresh fluent/fluent-bit:0.11.5-test docker image ? (updated minutes ago)

[dardymek]

Hi, now log looks like:

{
  "_index" : "logstash-2017.05.12",
  "_type" : "flb_type",
  "_id" : "AVv8UQxINSElbWyViQC7",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "@timestamp" : "2017-05-12T11:00:21",
    "log" : "{\\\"timestamp\\\":\\\"2017-04-18T08:51:50.275Z\\\",\\\"level\\\":\\\"DEBUG\\\",\\\"thread\\\":\\\"Camel (workflowUtils) thread #69 - seda://workflowSubmitCheckIfDefaultProjectNameProvided\\\",\\\"mdc\\\":{\\\"camel.exchangeId\\\":\\\"fc008063-c9c3-44bf-ae79-c1f05ba7f541\\\",\\\"camel.contextId\\\":\\\"workflowUtils\\\",\\\"camel.messageId\\\":\\\"cd23e6c6-2d93-4689-8721-57167cf2af1a\\\",\\\"camel.correlationId\\\":\\\"d614d241-88ac-4b87-b283-dfd09b89f5c1\\\",\\\"camel.breadcrumbId\\\":\\\"cd23e6c6-2d93-4689-8721-57167cf2af1a\\\",\\\"camel.routeId\\\":\\\"seda-workflow-submit-check-if-project-name-is-DefaultProjectName\\\"},\\\"logger\\\":\\\"com.intel.ccc.extension.vectorworkflow.process.userdata.UpdateDefaultProjectNameWithProjectId\\\",\\\"message\\\":\\\"WorkflowService: Project Id returned of Default Project is Map(project -\\u003e AVtYLpDjPCZbkFqZL4gt)\\\",\\\"context\\\":\\\"default\\\"}\\n",
    "stream" : "stdout",
    "time" : "2017-04-18T08:51:50.276272198Z",
    "kubernetes" : {
      "pod_name" : "ccc-frontend-3221816268-6q3tq",
      "namespace_name" : "default",
      "container_name" : "ccc-frontend",
      "docker_id" : "7e3eed60a5ed0a746def20d981fa273eb61a131e53481033287665767c79153e",
      "pod_id" : "9925a2fa-2064-11e7-8231-0290f4fa4413",
      "labels" : {
        "pod-template-hash" : "3221816268",
        "role" : "ccc-frontend"
      },
      "annotations" : {
        "kubernetes_io/created-by" : "{\\\"kind\\\":\\\"SerializedReference\\\",\\\"apiVersion\\\":\\\"v1\\\",\\\"reference\\\":{\\\"kind\\\":\\\"ReplicaSet\\\",\\\"namespace\\\":\\\"default\\\",\\\"name\\\":\\\"ccc-frontend-3221816268\\\",\\\"uid\\\":\\\"540c4fcf-1eab-11e7-8231-0290f4fa4413\\\",\\\"apiVersion\\\":\\\"extensions\\\",\\\"resourceVersion\\\":\\\"1064503\\\"}}\\n"
      }
    },
    "@fields" : {
      "timestamp" : "2017-04-18T08:51:50.275Z",
      "level" : "DEBUG",
      "thread" : "Camel (workflowUtils) thread #69 - seda://workflowSubmitCheckIfDefaultProjectNameProvided",
      "mdc" : {
        "camel_exchangeId" : "fc008063-c9c3-44bf-ae79-c1f05ba7f541",
        "camel_contextId" : "workflowUtils",
        "camel_messageId" : "cd23e6c6-2d93-4689-8721-57167cf2af1a",
        "camel_correlationId" : "d614d241-88ac-4b87-b283-dfd09b89f5c1",
        "camel_breadcrumbId" : "cd23e6c6-2d93-4689-8721-57167cf2af1a",
        "camel_routeId" : "seda-workflow-submit-check-if-project-name-is-DefaultProjectName"
      },
      "logger" : "com.intel.ccc.extension.vectorworkflow.process.userdata.UpdateDefaultProjectNameWithProjectId",
      "message" : "WorkflowService: Project Id returned of Default Project is Map(project -u003e AVtYLpDjPCZbkFqZL4gt)",
      "context" : "default"
    }
  }
}

Now index is created correctly but:

1. Too many \ in log field

\\\" are in log field. Should be \"

2. @fields, it was an example field, not mandatory one as parent. When in logs we have:

\"@fields\": { \"remote_addr\": \"172.17.0.1\", \"remote_user\": \"-\"}

we should have:

"_source" : {
        "@timestamp" : "2017-04-10T02:08:04+00:00",
        "@fields" : {
          "remote_addr" : "172.17.0.1",
          "remote_user" : "-"
        },

if we have:

\"remote_addr\": \"172.17.0.1\", \"remote_user\": \"-\"

I expect:

"_source" : {
        "@timestamp" : "2017-04-10T02:08:04+00:00",
        "remote_addr" : "172.17.0.1",
        "remote_user" : "-",

How correctly log from elastic generated by Fluentd looks like:

{
  "_index" : "logstash-2017.05.10",
  "_type" : "fluentd",
  "_id" : "AVv0YCWyRB7kL4qoiFIN",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "timestamp" : "2017-05-10T21:59:52.451Z",
    "level" : "DEBUG",
    "thread" : "ccc_agent-akka.actor.default-dispatcher-16357",
    "mdc" : {
      "sourceThread" : "ccc_agent-akka.actor.default-dispatcher-16355",
      "akkaTimestamp" : "21:59:52.451UTC",
      "akkaSource" : "akka://ccc_agent/user/agentHttpInterface",
      "sourceActorSystem" : "ccc_agent"
    },
    "logger" : "com.intel.ccc.agent.controller.JobControllerActor",
    "message" : "now watched by Actor[akka://ccc_agent/user/IO-HTTP/listener-0/91935#-902103998]",
    "context" : "default",
    "log" : "{\"timestamp\":\"2017-05-10T21:59:52.451Z\",\"level\":\"DEBUG\",\"thread\":\"ccc_agent-akka.actor.default-dispatcher-16357\",\"mdc\":{\"sourceThread\":\"ccc_agent-akka.actor.default-dispatcher-16355\",\"akkaTimestamp\":\"21:59:52.451UTC\",\"akkaSource\":\"akka://ccc_agent/user/agentHttpInterface\",\"sourceActorSystem\":\"ccc_agent\"},\"logger\":\"com.intel.ccc.agent.controller.JobControllerActor\",\"message\":\"now watched by Actor[akka://ccc_agent/user/IO-HTTP/listener-0/91935#-902103998]\",\"context\":\"default\"}\n",
    "stream" : "stdout",
    "docker" : {
      "container_id" : "e4bea946586d121a3138822030428627f8be6683b143d73f464d5786e7354126"
    },
    "kubernetes" : {
      "namespace_name" : "default",
      "pod_id" : "5bc79f77-30cd-11e7-8231-0290f4fa4413",
      "pod_name" : "ccc-agent-3200900416-l27sv",
      "container_name" : "ccc-agent",
      "labels" : {
        "pod-template-hash" : "3200900416",
        "role" : "ccc-agent"
      },
      "host" : "ip-10-0-2-88.us-west-2.compute.internal"
    },
    "@timestamp" : "2017-05-10T21:59:53.163+00:00",
    "tag" : "kubernetes.var.log.containers.ccc-agent-3200900416-l27sv_default_ccc-agent-e4bea946586d121a3138822030428627f8be6683b143d73f464d5786e7354126.log"
  }
}

[edsiper]

thanks for your feedback and patience.

I've taken in count your comments and the image have been updated with the following changes:

• when merge_json_log is enabled, all fields are inserted in the same level than 'log' field (no @fields)
• packager: auto-detect escaped string (do not quote everything)
• elasticsearch: new include_tag_key and tag_key options

Please pull a fresh copy of fluent/fluent-bit:0.11.5-test and let me know your comments.

[dardymek]

Thank you! It works great :)

"_index" : "logstash-2017.05.16",
      "_type" : "flb_type",
      "_id" : "AVwQ2yBNvHdGJ3mg3EDZ",
      "_score" : 3.8903718,
      "_source" : {
        "@timestamp" : "2017-05-16T10:43:34.%L",
        "log" : "{\"timestamp\":\"2017-05-16T10:43:25.358Z\",\"level\":\"DEBUG\",\"thread\":\"ccc_agent-akka.actor.default-dispatcher-36\",\"mdc\":{\"sourceThread\":\"ccc_agent-akka.actor.default-dispatcher-25\",\"akkaTimestamp\":\"10:43:25.358UTC\",\"akkaSource\":\"akka://ccc_agent/user/agentHttpInterface\",\"sourceActorSystem\":\"ccc_agent\"},\"logger\":\"com.intel.ccc.agent.controller.JobControllerActor\",\"message\":\"no longer watched by Actor[akka://ccc_agent/user/IO-HTTP/listener-0/847#489616481]\",\"context\":\"default\"}\n",
        "stream" : "stdout",
        "time" : "2017-05-16T10:43:25.359158656Z",
        "timestamp" : "2017-05-16T10:43:25.358Z",
        "level" : "DEBUG",
        "thread" : "ccc_agent-akka.actor.default-dispatcher-36",
        "mdc" : {
          "sourceThread" : "ccc_agent-akka.actor.default-dispatcher-25",
          "akkaTimestamp" : "10:43:25.358UTC",
          "akkaSource" : "akka://ccc_agent/user/agentHttpInterface",
          "sourceActorSystem" : "ccc_agent"
        },
        "logger" : "com.intel.ccc.agent.controller.JobControllerActor",
        "message" : "no longer watched by Actor[akka://ccc_agent/user/IO-HTTP/listener-0/847#489616481]",
        "context" : "default",
        "kubernetes" : {
          "pod_name" : "ccc-agent-3200900416-jb8xr",
          "namespace_name" : "default",
          "container_name" : "ccc-agent",
          "docker_id" : "d7354928981d6179c5181218c8024a1205e24e8972158373e6882d662eb52024",
          "pod_id" : "ca13690e-395b-11e7-9a64-022e2a654eba",
          "labels" : {
            "pod-template-hash" : "3200900416",
            "role" : "ccc-agent"
          },
          "annotations" : {
            "kubernetes_io/created-by" : "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicaSet\",\"namespace\":\"default\",\"name\":\"ccc-agent-3200900416\",\"uid\":\"d0b8d32c-1f77-11e7-8231-0290f4fa4413\",\"apiVersion\":\"extensions\",\"resourceVersion\":\"7862884\"}}\n"
          }
        }

The only thing left for me is how to add miliseconds to @timestamps. I opened new issue for that:
#2

[edsiper]

thanks.

These days I've been working on fixing the timestamp stuff, note that Fluent Bit 0.11 don't support milliseconds, but 0.12 (under dev) it does.

Fluent Bit 0.11.5 will be available shortly.

[edsiper]

@darek-dymek-i

would you mind to test the last time the 0.11.5-test image ?, I did some improvements and I want to check with you everything is perfect before the final release.

thanks for your help

[dardymek]

I downloaded last image and it looks ok. I'm waiting for release.

[edsiper]

thanks. Release is already available: fluent/fluent-bit:0.11.5

Release notes here:

http://fluentbit.io/announcements/v0.11.5/