isabella232 / sarkac

Apache Kafka auto-discovery anomaly detection :blowfish:

sarkac

Apache Kafka topic and message anomaly detection with automated discovery.

Install

Very simple via yarn add sarkac

Setup

Basically you can create your own apps with sarkac or integrate it in your existing apps, however you can also simply just spin up an instance by providing it some simple configuration info. Please note: that sarkac requires MongoDB 🌱 to store its windows.

You can find an example here.

Visit http://localhost:8033/ to check for sarkac's HTTP endpoints, that give infos about discovery and anomaly stats.

How does it work?

sarkac connects to your Kafka cluster and runs a simple discovery protocol to detect existing Kafka topics, it will automatically subscribe to them (also to newly added Kafka topics) and analyse their schema (has to be JSON), of their message payloads. It will then identify any fields of type 'number' and track them across all messages it receives. sarkac then uses MongoDB to keep multiple (as much as you configure) rolling windows of the values of the tracked fields. It runs the 68–95–99.7 rule on every window of every tracked field continously to detect anomalies. If an anomaly is detected it produces its information to an anomaly Kafka topic.

Running without auto-discovery

As shown in the example (uncommented dsl lines) it is also possible to deactivate auto discovery of topics and fields and simply run sarkac on fixed topics, by configuring the config.dsl object, do not forget to turn off discovery via config.discovery.enabled = false.

Additionally you can also turn off anomaly production to Kafka via config.target.produceAnomalies = false.

Use-case

Given a Kafka cluster with a certain amount of topics, keeping an eye on all of them at once can be challenging. And although we do not claim that you can cover all kinds of anomalies with sarkac, it can at least help you to tackle certain problems earlier. Just spin up a few instances and let them disover you Kafka broker's topics and produce anomalies to an output topic. Use our Kafka to Prometheus Connector to sink the anomaly topic into Prometheus and use Grafanas alert magic to get you notified based on detected anomalies.

Maintainer

Build with ❤️ 🍕 and ☕ by nodefluent