Producer terminate about broadway_kafka

[josevalim]

When you terminate the supervisor, by calling Broadway.stop, all processes under it will terminate. You only need to trap exits if you want to execute some logic in the terminate callback, which we don't need in the producer stage. How are you stopping the pipeline?

[FedericoPanS]

I am stopping the pipeline in the way you mentioned. In the KafkaBroadway.Producer the group coordinator is disconnected in the terminate callback which is called from the terminate callback in the ProducerStage. This callback is not called and some brod processes keep running after stopping the pipeline.

[josevalim]

Gotcha, so you are 100% correct. :) Would you like to send a PR to broadway_kafka? Then give it a try and let me know how it goes. I will move this issue there.

[FedericoPanS]

Thank you, I'll be happy to send a PR. Just to be sure, the way to go about it is to let each producer set the trap exit flag to true if they need to perform clean up in the terminate?
Also, is it possible to perform clean up in the processor? Right now I'm also sending to Kafka and would like to close the connection when the pipeline stops.
Thanks!

[josevalim]

The fix is to call Process.flag(:trap_exit, true) in the producer init callback. That will set it for each producer.

Broadway shutdown is graceful, so all of the code in the processor, batchers, etc will push the remaning work before terminating. There is no additional clean up on processors and batch processors at the moment but we could support this feature in the Broadway topology if necessary (i.e. its own terminate callback).