philips-software / fluent-bit-out-hsdp

fluent bit output plugin to HSDP logging

fluent bit HSDP logging output plugin

This plugin outputs your logs to the HSDP Host Logging service. This is useful when your workloads are not running on Cloud foundry, but you still want to utilize the central logging facilities of HSDP.

Fluent bit supports parser and filter plugin which can convert unstructured data gathered from the log Input interface into a structured one and to alter existing structured data before ingestion.

More on fluent-bit

Cloud and On-Premise

The plugin supports deployment to both Cloud and On-Premise environments. Depending on the deployment type you can either specify the Cloud Region and Environment or the On-Premise IamUrl and IdmUrl values.

API Signing, Service Identities or Logdrainer

The plugin supports both the API Signing authorization mechanism or the use of a IAM Service Identity with the LOG.CREATE scope. It also supports using a CF Logdrain endpoint although that limits the log format output (e.g. no support for the custom field)

Configuration options

Your fluent-bit.conf file should include an entry like below to enable the plugin:

[output]
    Name hsdp
    Match *

Configuring the authorization mechanism and HSDP Logging endpoints should ideally be done by setting the right Environment variables:

Settings

General

These keys are relevant when using either SigningKey or Service identities

Key	Description	Environment variable	Required
ProductKey	The Product key of your proposition	HSDP_PRODUCT_KEY	Required
Region	The HSP Region (Cloud)	HSDP_REGION	Required
Environment	THE HSP Environment (Cloud)	HSDP_ENVIRONMENT	Required
IamUrl	The IAM URL (On-Premise)	HSDP_IAM_URL	Optional
IdmUrl	The IDM URL (On-Premise)	HSDP_IDM_URL	Optional
IngestorHost	The HSDP ingestor host	HSDP_INGESTOR_HOST	Optional
Debug	Shows request details when set to true	HSDP_DEBUG	Optional
CustomField	Adds the field hash to custom field when set to true	HSDP_CUSTOM_FIELD	Optional
InsecureSkipVerify	Skip checking HSDP ingestor TLS cert. Insecure!	HSDP_INSECURE_SKIP_VERIFY	Optional
SynchronousFlush	Flushes log messages synchronously without batching. By default this is set to false		Optional
RetryOnError	Returns retry to FLB if flush fails. Applicable only when SynchronousFlush option is set. By default this is set to false		Optional

Signing keys

Key	Description	Environment variable	Required
SharedKey	The Shared key for signing requests	HSDP_SHARED_KEY	Optional
SecretKey	The Secret key for signing requests	HSDP_SECRET_KEY	Optional

Service identities

Key	Description	Environment variable	Required
ServiceId	The Service ID to use for authentication	HSDP_SERVICE_ID	Optional
ServicePrivateKey	The Service private key	HSDP_SERVICE_PRIVATE_KEY	Optional

Logdrain

You can reuse an existing Cloud foundry logdrainer endpoint to ship your logs to HSDP logging. The advantage is that you only need the Logdrain URL itself, no other configuration. This URL is considered a credential so care should be taken to protect it though.

Key	Description	Environment variable	Required
LogdrainUrl	The HSP Logdrain URL	HSDP_LOGDRAIN_URL	Optional
LogdrainApplicationName	The Application name to use	HSDP_LOGDRAIN_APPLICATION_NAME	Optional
LogdrainServerName	The Server name to use	HSDP_LOGDRAIN_SERVER_NAME	Optional

Environment variable values take precedence over those in configuration files.

Record field mapping to HSDP logging resource

The plugin supports full pass-through of the native LogEvent JSON message type, example:

{
  "resourceType": "LogEvent",
  "id": "7f4c85a8-e472-479f-b772-2916353d02a4",
  "applicationName": "OPS",
  "eventId": "110114",
  "category": "TRACELOG",
  "component": "TEST",
  "transactionId": "2abd7355-cbdd-43e1-b32a-43ec19cd98f0",
  "serviceName": "OPS",
  "applicationInstance": "INST‐00002",
  "applicationVersion": "1.0.0",
  "originatingUser": "SomeUsr",
  "serverName": "ops-dev.cloud.pcftest.com",
  "logTime": "2017-01-31T08:00:00Z",
  "severity": "INFO",
  "logData": {
    "message": "VGVzdCBNZXNzYWdl"
  },
  "custom": {
    "foo": "bar"
  }
}

NOTE: the logData.message field must be base64 encoded

field mapping

Alternatively, the plugin maps certain record fields to defined HSDP logging resource fields. The below table shows the mapping, and the default value.

Record field	HSDP logging field	Default value	Details
server_name	serverName	fluent-bit
app_name	applicationName	fluent-bit
app_instance	applicationInstance	fluent-bit
app_version	applicationVersion	1.0
category	category	TraceLog
severity	severity	informational
service_name	service_name	fluent-bit
originating_user	originating_user	fluent-bit
event_id	event_id	1
transaction_id	transaction_id	random UUID	if original input is not a valid UUID a new one will be generated
trace_id	trace_id
span_id	span_id
logdata_message	logData.Message	field hash	will replace the default field hash dump went present

Fields mapped to a HSDP logging resource field will be removed from the log message dump

The below filter definition shows an example of assigning fields

[filter]
    Name record_modifier
    Match *
    Record server_name ${HOSTNAME}
    Record service_name Awesome_Tool

[filter]
    Name modify
    Match *
    Rename container_name app_name
    Rename container_name service_name
    Rename component_name component
    Rename container_id app_instance

Remaining fields will be rendered to a JSON hash and assigned to logData.Message

Building

docker build -t fluent-bit-out-hsdp .

Testing with Docker

docker run --rm \
    -p 127.0.0.1:24224:24224 \
    -e HSDP_PRODUCT_KEY=product-key-here \
    -e HSDP_REGION=us-east \
    -e HSDP_ENVIRONMENT=client-test \
    -e [email protected] \
    -e HSDP_SERVICE_PRIVATE_KEY="$(cat service_private_key.pem)" \
    -it ghcr.io/philips-software/fluent-bit-out-hsdp:latest

Once the above is running you can start other Docker containers and use fluentd log driver to start logging to HSDP logging:

docker run --rm -it --log-driver fluentd alpine echo "hello world"

Helm Chart

You can deploy fluent-bit and the HSDP plugin using a Helm chart

Secret

The chart will attempt to read credentials from an hsdp-logging Kubernetes secret which should reside in the namespace. An example hsdp-logging-secret.yaml is included below. Make sure you replace the values accordingly.

NOTE: All the values should be in the base64 encoded.

apiVersion: v1
kind: Secret
metadata:
  name: hsdp-logging
type: Opaque
data:
  shared_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  secret_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  product_key: cmVwbGFjZV9tZV93aXRoX2NvcnJlY3RfdmFsdWVz
  ingestor_host: aHR0cHM6Ly9sb2dpbmdlc3RvcjItY2xpZW50LXRlc3QuZXUtd2VzdC5waGlsaXBzLWhlYWx0aHN1aXRlLmNvbQ==

Apply the secret to the right namepace (for Fiesta this is kube-system):

kubectl apply -f hsdp-logging-secret.yaml -n logging

Add helm repo:

helm repo add philips-software https://philips-software.github.io/helm-charts/

Download the helm chart

helm pull philips-software/fluent-bit-out-hsdp

Navigate to the download Helm chart and unzip.

After unzip, open fluent-bit-out-hsdp\values.yml and add the enviroment details like:

fluent-bit:
  env:
   - name: HSDP_REGION
     valueFrom:
       secretKeyRef:
         key: region
         name: hsdp-logging
         optional: true
   - name: HSDP_ENVIRONMENT
     valueFrom:
       secretKeyRef:
         key: environment
         name: hsdp-logging
         optional: true
   - name: HSDP_PRODUCT_KEY
     valueFrom:
       secretKeyRef:
         key: product_key
         name: hsdp-logging
         optional: false
   - name: HSDP_INGESTOR_HOST
     valueFrom:
       secretKeyRef:
         key: ingestor_host
         name: hsdp-logging
         optional: true
   - name: HSDP_SHARED_KEY
     valueFrom:
       secretKeyRef:
         key: shared_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SECRET_KEY
     valueFrom:
       secretKeyRef:
         key: secret_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SERVICE_ID
     valueFrom:
       secretKeyRef:
         key: service_id
         name: hsdp-logging
         optional: true
   - name: HSDP_SERVICE_PRIVATE_KEY
     valueFrom:
       secretKeyRef:
         key: service_private_key
         name: hsdp-logging
         optional: true

You can skip the unwanted fields from above yaml. For e.g, if you are going to use secret_key and shared_key based authentication, then only fields required are:

fluent-bit:
  env:
   - name: HSDP_PRODUCT_KEY
     valueFrom:
       secretKeyRef:
         key: product_key
         name: hsdp-logging
         optional: false
   - name: HSDP_INGESTOR_HOST
     valueFrom:
       secretKeyRef:
         key: ingestor_host
         name: hsdp-logging
         optional: true
   - name: HSDP_SHARED_KEY
     valueFrom:
       secretKeyRef:
         key: shared_key
         name: hsdp-logging
         optional: true
   - name: HSDP_SECRET_KEY
     valueFrom:
       secretKeyRef:
         key: secret_key
         name: hsdp-logging
         optional: true

Install the Helm chart

helm install my-fluent-bit-out-hsdp fluent-bit-out-hsdp -n logging

If the credentials are correct you should now see your Kubernetes cluster logs in the HSDP Logging system.

Contact / Getting help

Andy Lo-A-Foe [email protected]

License

License is MIT

fluent-bit-out-hsdp's Issues

Not able to generate new token for service identity after 30 minute expiry

Hi Andy,
I tested fluent bit plugin by providing service identity details. It works as expected for first 30 minutes. But after that it get stuck somewhere while getting token for service identity, there are no logs to console.
Looks like it’s not able to get new token for service identity after expiry.

Support for log-drainer capabilities as similar to CF bind service

In CF we were just binding deployed apps with the log drainer service (cups service to the HSP log host URL),
But now through fluent-bit-out-hsdp plugin and log ingestor approach we are forced to provide service identity detail or shared & secret key pair.

With logdrainer I mean, instead of using service identity or shared key/secrey, we would simply provide the logdrainer URL only as config.

InsecureSkipVerify and Debug variables

The current code checks for these variables either present or not. Based on that the action is taken care. However it would be nice, if code checks for boolean value, e.g. Debug=true then set else disable. Similarly for InsecureSkipVerify.
As we want to expose these variables in docker-compose so client can decide.

The default values for both can be false.

Github action for tagged releases

• Add Github Action for tagged releases

Update the parser for custom log event

The current custom_log_event in parsers.conf regex is not properly aligning the fields (attached screenshot). We use the below regex and it has proper output. can you take a look and update this?

[PARSER]
    Name   custom_log_event
    Format regex
    Regex  ^(?<severity>[^\|\s]+)\s*\|\s*CustomLogEvent\s*\|\s*(?<transaction_id>[^\|\s]*)\s*\|\s*(?<trace_id>[^\|\s]*)\s*\|\s*(?<span_id>[^\|\s]*)\s*\|\s*(?<component_name>[^\|\s]*)\s*\|\s*(?<logdata_message>.*)$

Service identity based authentication support

• Add support for service identity based auth

No valid credentials found with log drainer setting

We tried to configure the log drainer url to aggregate the logs, but when we try to run the command for build, we are getting "no valid credentials found".

we used the below settings:

docker run --rm
-p 127.0.0.1:24224:24224
-e HSDP_LOGDRAIN_URL=https://logdrainer-xxxx-healthsuite.com/core/log/Product/xxxxx
-it philipssoftware/fluent-bit-out-hsdp:latest