humio / kubernetes2humio

Ship Logs from Kubernetes to Humio

DEPRECATION

This repository has been deprecated in favour of the Fluent-bit chart

kubernetes2humio - Ship Logs from Kubernetes to Humio

Contains components for shipping logs and events from kubernetes clusters to Humio.

Overview

Fluentd is used to forward application- and host- level logs from each kubernetes node to a Humio server. This extends the standard setup from Fluentd for log forwarding in kubernetes. For clusters where the master nodes are not accessible (eg. on GCP) we use eventer to expose events occurring in the kubernetes control plane.

Quick Start

0. Pre-requisites:
   • Kubernetes cluster
   • User authorized to administrate via kubectl
   • Default service account with read privileges to API server for use by the kubernetes metadata filter plugin. This should be present by default in the kube-system namespace (even in kubernetes 1.6 with RBAC enabled)
1. Setup your data space in Humio and create an ingest-token
2. Base64 encode your ingest-token by running printf '<TOKEN>' | base64 and update fluentd/k8s/fluentd-humio-ingest-token-secret.yaml with the value
3. Update env. vars. FLUENT_HUMIO_HOST and FLUENT_HUMIO_DATA_SPACE in fluentd/k8s/fluentd-humio-daemonset.yaml
4. Create fluentd resources in kubernetes: kubectl apply -f fluentd/k8s/
5. Logs start appearing in Humio!

Optional:

6. If master nodes are not scheduleable in your cluster, you can also create eventer to expose control-plane events: kubectl apply -f eventer/
7. Add log-type pod labels to designate Humio parsers

Node-level Forwarding

In fluentd/docker-image/ a docker image is defined which specifies how to forward logs to Humio (with other settings, like log sources reused from the base image). Kubernetes manifests are defined in fluentd/k8s/: a daemonset will deploy fluentd pods across every worker node inside the kube-system namespace, and each pod will read the Humio ingest token from the fluentd-humio-ingest-token secret.

As per the normal setup, fluentd output is buffered, and uses TLS for nice log confidentiality. It also appends kubernetes metadata such as pod name and namespace to each log entry, wrapping raw logs in a standard json structure.

Log types

If your pod logs using JSON, Humio will parse the fields as excepted. If your logs are text based, e.g. an nginx access log, you can set the label log-type on a pod. Humio will use the log-type label to determine which parser to apply to the log line. Using a parser you can retain the structure in the logs. If the label is unspecified or doesn't correspond to a parser then pod logs will be left as unstructured text.

Fluentd Container Variables

We expose three environment variables so the daemonset configuration can be easily changed in different environments:

• FLUENT_HUMIO_HOST: Humio host
• FLUENT_HUMIO_DATA_SPACE: your data space
• FLUENT_HUMIO_INGEST_TOKEN: authorization to push logs into Humio

If you need to make further customizations, you will need to mount in an altered version of the fluentd config files /fluentd/etc/fluent.conf and /fluentd/etc/kubernetes.conf, e.g. using ConfigMaps.

Namespacing and Service Accounts Usage

As noted above, the 'default' service account is used by the fluentd metadata plugin to lookup pod/namespace information. This is not particularly in line with the developing RBAC model for service accounts in kubernetes, but causes few problems in the kube-system namespace where services are assumed to be somewhat root-like. Since 'default' service account is available to all pods in a namespace, careful thought is recommended when assigning permissions to this account to get fluentd to work outside the kube-system namespace.

Control-plane Events

Appropriate for clusters where fluentd cannot run on master nodes, the eventer component of heapster is used to retrieve cluster events from the API server. We forward events to fluentd by simply printing events to stdout, providing a consistent interface for logs coming out of kubernetes. Eventer runs as a deployment with a single instance handling all cluster events, regardless of cluster size. As with heapster, it makes use of the addon-resizer component to update requested resources as load on the eventer, causing the eventer pod to get redeployed as cluster activity grows past certain thresholds.

What about metrics?

We are currently working on integrating metrics from heapster into Humio. Stay tuned...

kubernetes2humio's Issues

Seems not to work on GKE

Hey all,

I tried to use this integration in my GKE cluster yesterday (Kubernetes 1.9.2) but couldn't see any logs flowing into my Humio dataspace.

Below is the log I take from the fluentd pod (I replaced the token and dataspace with fake values for privacy):

2018-02-28 18:24:56 +0000 [info]: reading config file path="/fluentd/etc/fluent.conf"
2018-02-28 18:24:56 +0000 [info]: starting fluentd-0.12.33
2018-02-28 18:24:56 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.9.5'
2018-02-28 18:24:56 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.27.0'
2018-02-28 18:24:56 +0000 [info]: gem 'fluent-plugin-record-modifier' version '0.6.0'
2018-02-28 18:24:56 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.0'
2018-02-28 18:24:56 +0000 [info]: gem 'fluent-plugin-secure-forward' version '0.4.5'
2018-02-28 18:24:56 +0000 [info]: gem 'fluentd' version '0.12.33'
2018-02-28 18:24:56 +0000 [info]: adding match pattern="fluent.**" type="null"
2018-02-28 18:24:56 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2018-02-28 18:25:00 +0000 [info]: adding filter pattern="**" type="record_modifier"
2018-02-28 18:25:00 +0000 [info]: adding filter pattern="**" type="grep"
2018-02-28 18:25:00 +0000 [info]: adding filter pattern="**" type="record_transformer"
2018-02-28 18:25:00 +0000 [info]: adding match pattern="**" type="elasticsearch"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: adding source type="tail"
2018-02-28 18:25:00 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    type null
  </match>
  <source>
    type tail
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag kubernetes.*
    read_lines_limit 500
    format json
    read_from_head true
  </source>
  <source>
    type tail
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
  </source>
  <source>
    type tail
    format syslog
    path /var/log/startupscript.log
    pos_file /var/log/fluentd-startupscript.log.pos
    tag startupscript
  </source>
  <source>
    type tail
    format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
    path /var/log/docker.log
    pos_file /var/log/fluentd-docker.log.pos
    tag docker
  </source>
  <source>
    type tail
    format none
    path /var/log/etcd.log
    pos_file /var/log/fluentd-etcd.log.pos
    tag etcd
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kubelet.log
    pos_file /var/log/fluentd-kubelet.log.pos
    tag kubelet
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-proxy.log
    pos_file /var/log/fluentd-kube-proxy.log.pos
    tag kube-proxy
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-apiserver.log
    pos_file /var/log/fluentd-kube-apiserver.log.pos
    tag kube-apiserver
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-controller-manager.log
    pos_file /var/log/fluentd-kube-controller-manager.log.pos
    tag kube-controller-manager
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/kube-scheduler.log
    pos_file /var/log/fluentd-kube-scheduler.log.pos
    tag kube-scheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/rescheduler.log
    pos_file /var/log/fluentd-rescheduler.log.pos
    tag rescheduler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/glbc.log
    pos_file /var/log/fluentd-glbc.log.pos
    tag glbc
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <source>
    type tail
    format kubernetes
    multiline_flush_interval 5s
    path /var/log/cluster-autoscaler.log
    pos_file /var/log/fluentd-cluster-autoscaler.log.pos
    tag cluster-autoscaler
    format_firstline /^\w\d{4}/
    format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
    time_format %m%d %H:%M:%S.%N
  </source>
  <filter kubernetes.**>
    type kubernetes_metadata
  </filter>
  <filter **>
    type record_modifier
    remove_keys log,kubernetes
    <record>
      message [${record['kubernetes']['namespace_name']}/${record['kubernetes']['container_name']}] ${record['log']}
      k8s ${record['kubernetes']}
      disable ${record['kubernetes']['labels']['disable-logging']}
    </record>
  </filter>
  <filter **>
    @type grep
    exclude1 disable ^true$
  </filter>
  <filter **>
    @type record_transformer
    remove_keys disable
  </filter>
  <match **>
    type elasticsearch
    include_tag_key false
    host cloud.humio.com
    path /api/v1/dataspaces/SPACE/ingest/elasticsearch/
    scheme https
    port 443
    user TOKEN
    password xxxxxx
    logstash_format true
    reload_connections true
    logstash_prefix fluentd:kubernetes2humio
    buffer_chunk_limit 1M
    buffer_queue_limit 32
    flush_interval 1s
    max_retry_wait 30
    disable_retry_limit 
    num_threads 8
  </match>
</ROOT>
2018-02-28 18:25:00 +0000 [info]: following tail of /var/log/containers/fluentd-jqck5_kube-system_fluentd-ced36e855a628b6b616586f667d9e426c412de7bcb4fad45d79beaf57a2ea75f.log
2018-02-28 18:25:03 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" tag="kubernetes.var.log.containers.fluentd-jqck5_kube-system_fluentd-ced36e855a628b6b616586f667d9e426c412de7bcb4fad45d79beaf57a2ea75f.log"
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-record-modifier-0.6.0/lib/fluent/plugin/filter_record_modifier.rb:156:in `expand'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-record-modifier-0.6.0/lib/fluent/plugin/filter_record_modifier.rb:92:in `block (2 levels) in filter_stream'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-record-modifier-0.6.0/lib/fluent/plugin/filter_record_modifier.rb:91:in `each_pair'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-record-modifier-0.6.0/lib/fluent/plugin/filter_record_modifier.rb:91:in `block in filter_stream'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event.rb:130:in `block in each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event.rb:129:in `each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event.rb:129:in `each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-record-modifier-0.6.0/lib/fluent/plugin/filter_record_modifier.rb:90:in `filter_stream'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event_router.rb:152:in `block in emit'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event_router.rb:151:in `each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event_router.rb:151:in `emit'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/event_router.rb:90:in `emit_stream'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:301:in `receive_lines'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:419:in `wrap_receive_lines'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:614:in `on_notify'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:445:in `on_notify'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:425:in `attach'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:198:in `setup_watcher'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:216:in `block in start_watchers'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:203:in `each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:203:in `start_watchers'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:192:in `refresh_watchers'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/plugin/in_tail.rb:135:in `start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/root_agent.rb:115:in `block in start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/root_agent.rb:114:in `each'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/root_agent.rb:114:in `start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/engine.rb:237:in `start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/engine.rb:187:in `run'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:570:in `run_engine'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:162:in `block in start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:366:in `main_process'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:339:in `block in supervise'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:338:in `fork'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:338:in `supervise'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/supervisor.rb:156:in `start'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-02-28 18:25:03 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.33/bin/fluentd:5:in `<top (required)>'
  2018-02-28 18:25:03 +0000 [warn]: /usr/bin/fluentd:22:in `load'
  2018-02-28 18:25:03 +0000 [warn]: /usr/bin/fluentd:22:in `<main>'
2018-02-28 18:25:03 +0000 [info]: following tail of /var/log/containers/calico-node-m42zb_kube-system_calico-node-38efb2dbd73ded69938e9378f7e85909ec5500370174af28bf56684261e9bc0d.log
2018-02-28 18:25:06 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" tag="kubernetes.var.log.containers.calico-node-m42zb_kube-system_calico-node-38efb2dbd73ded69938e9378f7e85909ec5500370174af28bf56684261e9bc0d.log"

Once the following warn message starts kicking, it's the only thing that I see:

2018-02-28 18:25:06 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `[]' for nil:NilClass" tag="kubernetes.var.log.containers.calico-node-m42zb_kube-system_calico-node-38efb2dbd73ded69938e9378f7e85909ec5500370174af28bf56684261e9bc0d.log"

for various sources.

In order to deploy properly, I used the Kube specs from this repository and added the following one:

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: humio-clusterrole
rules:
- apiGroups:
  - ""
  resources:
  - pods  
  verbs:
  - list
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: humio-clusterrole-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: humio-clusterrole
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-system

So the pod could request the list of pods.

I think the base image this repository is using is quite old (fluentd 0.12.33) while the latest is 1.x. This warning message above seemed to also have been worked on since 0.14.

There is an request to bump the base image as well but I feel it won't happen anytime soon unfortunately.

So, either I'm missing something entirely and I'd love to know what. Or this integration does not work with GKE and this make it worthless for now.