Question: How do I consume from multiple topics? about kafka-streams

[paambaati]

For anyone that ended up here looking for a similar solution, here's how I ended up doing it —

import { KafkaStreams } from 'kafka-streams';

const kafkaStreams = new KafkaStreams({...});

const inputTopics = ['t1', 't2'];
const inputStreams = inputTopics.map(inputTopic => {
    return kafkaStreams.getKStream(inputTopic);
});
const mergedStream = inputStreams.reduce((allStreams, stream) => {
    return allStreams.merge(stream);
});

mergedStream.forEach(message => console.log);

Promise.all(inputStreams.map(inputStream => {
    return inputStream.start();
}));

@krystianity Is this the recommended method of consuming from multiple topics?

[krystianity]

Hi @paambaati the recommended way is to use myKStream.from(["one", "two"]) :) alternatively myKStream.from("one").from("two") should also work.

[paambaati]

@krystianity Thanks for the tip! I tried it, and this prints messages from only 1 topic —

import { KafkaStreams } from 'kafka-streams';

const kafkaStreams = new KafkaStreams({...});

const inputTopics = ['t1', 't2'];
const inputStream = kafkaStreams.getKStream(null).from(inputTopics);

inputStream.forEach(message => console.log);

inputStream.start();

Is there more info I can provide?

[krystianity]

there are now 3 examples on how this works :)