Question about :receive_interval and topology. about broadway_kafka

[josevalim]

They go through the producer.

receive interval is the frequency to ping when the queue is empty. It doesn’t impact when the queue is full. It should not generate additional work to consumers.

[lud]

Ok José, thank you :)

[lud]

Hi again @josevalim

I found that if y set the concurrency option at the producer level to 5, messages are delivered instantly. The default 1 has the long polling delays, even with receive_interval set to 1.

This happens when all topics are fully committed, (queues are empty), I produce a message to Kafka and expect a "response" message to be delivered to the Broadway producer.

For now I keep it as-is but it feels weird to use multiple producers.

[lud]

I tried an alternative implementation with Kaffe and I found the same behaviour, this time because of the sleep_timeout option for brod.

Though I guess broadway_kafka uses fetch instead of subscribers so I guess that is another problem.