muhlba91 / telegraf-output-kinesis-data-firehose

Telegraf Output Plugin for Amazon Kinesis Data Firehose

Telegraf Output Plugin for Amazon Kinesis Data Firehose

---

⚠️ Attention: first, and foremost, this plugin is developed for personal use. In fact, it is not - and will not be - fully production ready!

---

This plugin makes use of the Telegraf Output Execd plugin. It will batch up Points in one Put request to Amazon Kinesis Data Firehose.

The plugin also provides optional common formatting options, like normalizing keys and flattening the output. Such configuration can be used to provide data ingestion without the need of a data transformation function.

It expects that the configuration for the output ship data in line format.

---

About Amazon Kinesis Data Firehose

It may be useful for users to review Amazons official documentation which is available here.

Usage

• Download the latest release package for your platform.

• Unpack the build to your system:

mkdir /var/lib/telegraf/firehose
chown telegraf:telegraf /var/lib/telegraf/firehose
tar xf telegraf-output-kinesis-data-firehose-<LATEST_VERSION>-<OS>-<ARCH>.tar.gz -C /var/lib/telegraf/firehose
# e.g. tar xf telegraf-output-kinesis-data-firehose-v1.0.0-linux-amd64.tar.gz -C /var/lib/telegraf/firehose

• Edit the plugin configuration as needed:

vi /var/lib/telegraf/firehose/plugin.conf

• Add the plugin to /etc/telegraf/telegraf.conf or into a new file in /etc/telegraf/telegraf.d:

[[outputs.execd]]
  command = [ "/var/lib/telegraf/firehose/telegraf-output-kinesis-data-firehose", "-config", "/var/lib/telegraf/firehose/plugin.conf" ]
  data_format = "influx"

• Restart or reload Telegraf.

AWS Authentication

This plugin uses a credential chain for Authentication with the Amazon Kinesis Data Firehose API endpoint. The plugin will attempt to authenticate in the following order:

1. web identity provider credentials via STS if role_arn and web_identity_token_file are specified,
2. assumed credentials via STS if the role_arn attribute is specified (source credentials are evaluated from subsequent rules),
3. explicit credentials from the access_key, and secret_key attributes,
4. shared profile from the profile attribute,
5. environment variables,
6. shared credentials, and/or
7. the EC2 instance profile.

If you are using credentials from a web identity provider, you can specify the session name using role_session_name. If left empty, the current timestamp will be used.

AWS IAM Policy

The required AWS IAM Policy is:

{
    "Statement": [
        {
            "Action": [
                "firehose:PutRecordBatch",
                "firehose:PutRecord"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:firehose:eu-west-1:126125163971:deliverystream/aws-firehose-homeassistant-stream-c49c07d",
            "Sid": ""
        },
        {
            "Action": "firehose:DescribeDeliveryStream",
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

Configuration

See the file plugin.conf as an example configuration file.

---

Notes

The plugin was inspired by the Amazon Kinesis Data Stream Output Plugin.

Supporting

If you enjoy the application and want to support my efforts, please feel free to buy me a coffe. :)

telegraf-output-kinesis-data-firehose's Issues

execd instead of exec

an execd plugin will be more performant since the startup time will be reduced.
is it (easily) possible to make the plugin execd instead of exec compatible?

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

github-actions

.github/workflows/release.yml

• google-github-actions/release-please-action v4
• actions/checkout v4
• actions/setup-go v5
• sigstore/cosign-installer v3.5.0
• anchore/sbom-action v0.17.0
• goreleaser/goreleaser-action v6

.github/workflows/verify.yml

• actions/checkout v4
• actions/setup-go v5
• actions/checkout v4
• actions/setup-go v5
• golangci/golangci-lint-action v6
• actions/checkout v4
• actions/setup-go v5
• shogo82148/actions-goveralls v1
• actions/checkout v4
• hashicorp/vault-action v3
• github/codeql-action v3

gomod

go.mod

• go 1.22.0
• go 1.22.5
• github.com/aws/aws-sdk-go-v2 v1.30.3
• github.com/aws/aws-sdk-go-v2/service/firehose v1.32.0
• github.com/influxdata/telegraf v1.31.2
• github.com/stretchr/testify v1.9.0

---

• Check this box to trigger a request for Renovate to run again on this repository