Offsets accumulating in the producer ack state about broadway_kafka

[josevalim]

Nothing obvious come to mind, unfortunately.

[danmarcab]

Thank you for having a look @josevalim, really appreciated!

After debugging for another while we have found one of the possible causes (I say one because we've seen offsets accumulating without this error message) of this, the batcher dies due to an unknown timer.

"GenServer MyService.Broadway.Broadway.Batcher_ignore terminating
    ** (RuntimeError) unknown timer #Reference<0.1181658914.3223060481.95258>
        (broadway 1.0.3) lib/broadway/topology/batcher_stage.ex:207: Broadway.Topology.BatcherStage.cancel_batch_timeout/1
        (broadway 1.0.3) lib/broadway/topology/batcher_stage.ex:148: Broadway.Topology.BatcherStage.deliver_batch/6
        (broadway 1.0.3) lib/broadway/topology/batcher_stage.ex:118: Broadway.Topology.BatcherStage.handle_events_per_batch_key/3
        (broadway 1.0.3) lib/broadway/topology/batcher_stage.ex:64: anonymous fn/2 in Broadway.Topology.BatcherStage.handle_events/3
        (telemetry 1.1.0) /build/deps/telemetry/src/telemetry.erl:320: :telemetry.span/3
        (broadway 1.0.3) lib/broadway/topology/batcher_stage.ex:54: Broadway.Topology.BatcherStage.handle_events/3
        (gen_stage 1.1.2) lib/gen_stage.ex:2471: GenStage.consumer_dispatch/6
        (gen_stage 1.1.2) lib/gen_stage.ex:2660: GenStage.take_pc_events/3

It seems that Broadway already accounts for the case where the timeout message has been received when the cancel timer returns false (https://github.com/dashbitco/broadway/blob/main/lib/broadway/topology/batcher_stage.ex#L201-L213), but there seems to be an edge case?

My guess is crashing the batcher is fine for other Producers where the ACK is not sequential, but for BroadwayKafka it seems like it's messing up.

[josevalim]

I am looking at the code and I cannot see a code path that would make the error message above happen. Every time we cancel the timer, we delete the batch, which means it is impossible to recover the timer again.

[josevalim]

This has been fixed in Broadway. There was an assumption that the timer message would be delivered automatically but that was not always the case.

[danmarcab]

This has been fixed in Broadway. There was an assumption that the timer message would be delivered automatically but that was not always the case.

This is great news! Thank you very much for all your work @josevalim ❤️