How to connect with elastic cloud? about fluent-bit-kubernetes-logging

[elpablete]

Besides using tls On make sure not to include the protocol (https://) part in your FLUENT_ELASTICSEARCH_HOST. At least that was my issue.

[ledroide]

@xncbf : I have a similar issue. Do you solve yours ?
Here is the fluent bit documentation about elasticsearch output plugin, it looks like you have the right parameters.

Here is my log output from a random fluent-bit 1.3.3 daemonset :

[2019/11/29 12:44:51] [error] [io fd=50] error sending data to: elasticsearch-es-http:9200 (Broken pipe)
[2019/11/29 12:44:51] [error] [src/flb_http_client.c:844 errno=25] Inappropriate ioctl for device
[2019/11/29 12:44:51] [ warn] [out_es] http_do=-1 URI=/_bulk
[2019/11/29 12:46:28] [error] [src/flb_http_client.c:844 errno=32] Broken pipe
[2019/11/29 12:46:28] [ warn] [out_es] http_do=-1 URI=/_bulk
[2019/11/29 12:51:40] [error] [io fd=50] error sending data to: elasticsearch-es-http:9200 (Broken pipe)
[2019/11/29 12:51:40] [error] [src/flb_http_client.c:844 errno=25] Inappropriate ioctl for device
[2019/11/29 12:51:40] [ warn] [out_es] http_do=-1 URI=/_bulk

And my conf :

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}

[luislhl]

@ledroide , did you find out the solution?

I was having the same problem as you, which I fixed by enabling tls on the output configuration.

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Logstash_Format On
        Replace_Dots    On
        Retry_Limit     False
        tls             On

The errors stopped being logged, by I still couldn't make fluent-bit send anything to my Elasticsearch hosted on Elastic Cloud.

I'm having this output now on fluent-bit containers after adding tls:

Fluent Bit v1.3.11
Copyright (C) Treasure Data

[2020/04/23 18:52:17] [ info] [storage] version=1.0.3, initializing...
[2020/04/23 18:52:17] [ info] [storage] in-memory
[2020/04/23 18:52:17] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/04/23 18:52:17] [ info] [engine] started (pid=1)
[2020/04/23 18:52:17] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/04/23 18:52:17] [ info] [filter_kube] local POD info OK
[2020/04/23 18:52:17] [ info] [filter_kube] testing connectivity with API server...
[2020/04/23 18:52:17] [ info] [filter_kube] API server connectivity OK
[2020/04/23 18:52:17] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2020/04/23 18:52:17] [ info] [sp] stream processor started

UPDATE

Actually the logs are working for me now.

The only thing left for me was to create the Index Pattern in Kibana, so I could visualize the logs. I created a Index Pattern for logstash-*, and can see the logs now. But I don't why exactly they are being created in that index, since the default is fluentbit.

But I also think @ledroide 's problem is not http, as I said before. I tested and it also worked.

[ckgmg]

Change the following to solve the problem

fluent-bit-configmap.yaml

output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
Logstash_Format On
Logstash_Prefix k8s-fluentbit
Replace_Dots On
Retry_Limit False
tls On
tls.verify Off
Type flb

fluent-bit-ds.yaml
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elastic-es-http.logging.svc"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "https"
- name: FLUENT_ELASTICSEARCH_USER
value: "elastic"
- name: FLUENT_ELASTICSEARCH_PASSWORD
value: "*****"