A way to access brod client about broadway_kafka

[chulkilee]

Since brod takes atom client id, you may pass the brod client id to broadway and use the same id from your module.

See (see https://github.com/dashbitco/broadway_kafka/blob/v0.1.2/lib/kafka_client.ex#L6 - you can pass the client_id option to broadway.

[josevalim]

Thanks for the help @chulkilee !

[escobera]

Hello @chulkilee , thanks a lot for the help!

I tried what you suggested, but was getting {:error, :client_down} errors from brod. So I noticed that even after I set up the client_id config it was still using an auto-generated name like this:

[info] [supervisor: {:local, :brod_sup}, started: [pid: #PID<0.2162.0>, id: PropartesPje.Pje1i.BroadwayPje1i.Broadway.Producer_0.Client, mfargs: {:brod_client, :start_link, [[localhost: 9092], PropartesPje.Pje1i.BroadwayPje1i.Broadway.Producer_0.Client, []]}, restart_type: {:permanent, 10}, shutdown: 5000, child_type: :worker]]

My broadway config:

def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {BroadwayKafka.Producer,
           [
             hosts: Application.get_env(:propartes_pje, :kafka_host, localhost: 9092),
             group_id: "grupo-ex-propartes",
             client_id: :my_test_client,
             topics: ["postgres-pje"],
             client_config: Application.get_env(:propartes_pje, :kafka_client_config, [])
           ]},
        concurrency: 1
      ],
      processors: [
        default: [
          concurrency: 1
        ]
      ]
    )
  end

I think the client_id is being changed here:

broadway_kafka/lib/producer.ex

Line 171 in a1ee339

client_id = Module.concat([producer_name, Client])

[msaraiva]

Hi @escobera!

We could allow the user to set the client_id explicitly but I'm not sure that sharing it would be a good idea. The problem is that each producer process is responsible for the lifecycle of its own client so they should be considered "private" as you can't be sure when it would be available or whether it's being restarted or not. So I think it would be safer to keep them separate. I wouldn't worry much about performance either. It shouldn't add any significant overhead.

[escobera]

Gotcha, thanks for the help!