authorjapps / hello-kafka-stream-testing

The most simple way to test Kafka based applications or micro-services e.g. Read/Write during HBase/Hadoop or other Data Ingestion Pipe Lines

Kafka Testing Hello World examples

This repo used open-source lib zerocode-tdd for declarative style testing. Many flavours of HelloWorld samples are available to clone and run.

Please make sure you bring up Kafka in a Docker prior to running the tests.

Let's learn the most simple and efficient way of automated testing of Kafka applications. This is particulaly useful during:

• Micro-Services involving REST and Kafka
• Read/Write during BigData store e.g. HBase/Hadoop
• any other Data Pipe Lines

Try-at-home examples and much more(click to exapnd)

• Kafka testing - Examples to run at home

• Kafka testing - An Intro

• Other HelloWorld examples, such as Spring boot app testing, Performance testing, Kotlin app testing etc.

For running the below test, please jump to the corresponding JUnit @Test.

@TargetEnv("kafka_servers/kafka_test_server.properties")
@RunWith(ZeroCodeUnitRunner.class)
public class KafkaProduceTest {

    @Test
    @JsonTestCase("kafka/produce/test_kafka_produce.json")
    public void testProduce() throws Exception {
        // No code needed here.
    }

}

In the above code

• test_kafka_produce.json is the Test Case which contains the JSON step(s). See a sample below.
• kafka_test_server.properties contains the "Broker" details and Producer/Consumer configs
• @RunWith(ZeroCodeUnitRunner.class) is a JUnit custom runner to run the test

e.g.

{
    "scenarioName": "Simple Produce and Consume  a record to-from a kafka topic",
    "steps": [
        {
            "name": "produce_step",
            "url": "kafka-topic:demo-topic1",
            "operation": "produce",
            "request": {
                "records":[
                    {
                        "key": "${RANDOM.NUMBER}",
                        "value": "Hello World"
                    }
                ]
            },
            "assertions": {
                "status" : "Ok"
            }
        },
        {
            "name": "consume_step",
            "url": "kafka-topic:demo-topic1",
            "operation": "consume",
            "request": {
            },
            "assertions": {
                "size": 1,
                "records": [
                    {
                        "key" : "$NOT.NULL",
                        "value": "Hello World"
                    }
                ]
            }
        }
    ]
}

Now you can see the-

• Reports @ target
• Logs @ target/logs/test_logs.log
• Test coverage CSV Report @ target/zerocode-junit-granular-report.csv
• Test coverage Chart @ target/zerocode-junit-interactive-fuzzy-search.html
• More reports

IDE References:

Eclipse(General key-board shotcuts):

For quick reference only - See more eclipse keys

1. Open a matching java file -> Ctrl + Shift + R
2. Open a matching JSON file -> Ctrl + Shift + R
3. To navigate to file -> Ctrl + Click
4. Clik a JSON file and navigate to it's JUnit Test -> Alt+Cmd+G(Mac: ⌥+⌘+G), Windows(Alt+Ctrl+G)

IntelliJ(General key-board shotcuts):

More keys - See more IntelliJ keys

1. Open a matching java file -> Ctrl + n
2. Open a matching JSON or XML file -> Ctrl + Shift + n
3. To navigate to file -> Ctrl + Click
4. Usage/invocation of a JSON file or Java File or Java method/variable etc -> Alt + F7

hello-kafka-stream-testing's Issues

kafka streams headers in the produce step

I think it is not possible to define headers in the producer json. Something like this:

"steps": [
    {
      "name": "produce_step",
      "url": "kafka-topic:xxxx",
      "operation": "produce",
      "request": {
        "records": [
          {
            **"headers": [
               {"type": "aaa"}
            ],**
            "key": "577152c4-26d2-4479-ae05-a7272fa81eea",
            "value": "{\"eventId\":\"577152c4-26d2-4479-ae05-a7272fa81eea\"}"
          }
        ]
      },
      "assertions": {
        "status": "Ok"
      }

Do you know any similar python package/API for testing Kafka like this?

ZeroCodeUnitRunner does not work with JUnit5 Jupiter tests

Hi there! Followed your example here: https://github.com/authorjapps/hello-kafka-stream-testing but I am using JUnit5 in my project. The tests fail with No runnable methods error. I tried including the zerocode-tdd-jupiter module as well in my dependencies, but it fails with same error. Any help would be appreciated!