Multiple processors when having a single partition/producer about broadway_kafka

[josevalim]

@epilgrim kafka expects messages to be consumed in order because they can only be acknowledge in order (i.e. you can only acknowledge message 10 after you consumed message 9). So I can't see anyway for you to leverage concurrency without dropping the ack guarantees... which I wouldn't recommend doing?

[juanperi]

yeah, I understand that.
I was hoping that I could use batching in some way to be able to do it.
Imagine that processing 1 message takes 1 second. If I have 20 messages, it will take 20 seconds to finish.

Now, imagine that I can process 10 messages at a time, then broadway will bring 10 messages in a negligible time, then 10 processors work 1 second each. I would have consumed (given that there were no errors) everything in 2 seconds.

In the case that there were errors, we would need to respect the original order for ACK. I don't want to drop those guarantees either.

I guess I was expecting (hoping, in fact) there was some kind of batching that allowed me to consume 10 messages at a time, and ACK them all at the same time. Or even better, a way for the ACK to keep the order, and not send ACKs if the previous offsets have not been processed successfully.

In any case, it's not a major issue. I can keep the whole processing in a serialized way.
Please close the issue if there is no further clarification I can make on the case at hand

Thank you!!!

[josevalim]

If you process message 10 before process 9 because of concurrency and then processing 9 fails, you will have to process message 10 again.

I can think of two options for your case:

1. You can use the prepare_messages from Broadway master to do some parallel computation on the prepare stage
2. Get rid of processors and do everything in the batching stage so you can use task.async_stream or similar in there

But overall, I think the behaviour is correct within Kafka constraints. :)