Home

Kafka-Penguin

An easy-to-use, lightweight KafkaJS library for message re-processing.

About
Getting Started
Example
Strategies
Contributors
License

About

Kafka-Penguin is an easy-to-use, lightweight KafkaJS plugin for message re-processing. It provides developers with three strategies for setting up message re-processing: FailFast, Ignore, and Dead Letter Queue.‌

The package allows developers to build event-driven applications with dedicated "fail strategies" modeled after best practices in the field. This in turn allows developers to effectively address bugs in development and deploy more fault-tolerant systems in production.‌

This package is meant to work in conjunction with with KafkaJS. For more information on KafkaJS, check out Getting Started with KafkaJS.‌

Getting Started

Install Kafka-Penguin as an npm module and save it to your package.json file as a dependency:

npm install kafka-penguin

Once installed it can now be referenced by simply calling require('kafka-penguin');‌

Example

All Kafka-Penguin needs is a KafkaJS client to run. Start by passing the client for your preferred strategy and Kafka-Penguin will create bespoke consumers, producers, and admins with built-in functionality to execute the chosen strategy. On the surface, you implement your application exactly as you would with KafkaJS.

/* eslint-disable no-console */
import { FailFast } from 'kafka-penguin';

const exampleClient = require('./clientConfig.ts');

// Set up the preferred strategy with a configured KafkaJS client
const exampleStrategy = new FailFast(2, exampleClient);

// Initialize a producer or consumer from the instance of the strategy
const producer = exampleStrategy.producer();

const message = {
  topic: 'wrong-topic',
  messages: [
    {
      key: 'hello',
      value: 'world',
    },
  ],
};

// Connect, Subscribe, Send, or Run virtually the same as with KafkaJS
producer.connect()
  .then(() => console.log('Connected!'))
  // The chosen strategy executes under the hood, like in this send method
  .then(() => producer.send(message))
  .catch((e: any) => console.log('error: ', e.message));