KafkaProducerBundle and Topics about dropwizard-kafka

[natalie-zamani]

Hey there @michael-forman!

If I understand the lifecycle of bundles etc, doesn't this mean that the topics can't come from Configuration?

It does mean that currently, yes.

Am I missing something by wanting to retrieve the topics from configuration?

No, I think that's reasonable, and it's a bit of an oversight on my part. If you'd like to submit a PR to allow specifying topics via configuration, while maybe allowing topic overrides to still be provided via constructor (for backwards-compatibility), I'd be happy to review it! I'm also happy to set aside some time this weekend to make the change myself if not too.

[michael-forman]

Hi @mzamani1,

Thanks for the reply.

I'm not sure whether the Topic config belongs in KafkaProducerFactory class, or elsewhere.

I was looking at (and did in a "fork" of KafkaProducerBundle I made in my project) perhaps adding something like:
public abstract Collection<String> getKafkaTopics(T configuration);

into the KafkaProducerBundle, similar concept to getKafkaProducerFactory. The call is late enough in the lifecycle that Configuration is available. The client subclass of KafkaProducerBundle then decides how to get that list - they could still hard code, or get it from config, just without any pre-made configuration class.

The issue with keeping topics in the contstructor, and adding another mechanism to get them is that I'm not sure what the right semantics are if topics were already provided in the constructor and then more/different ones added via that callback. The code could not call getKafkaTopics if there were any topics passed in at construction, or it could add to the existing.

Or there is possibly (probably) a better way entirely to achieve the desired outcome...

I'm happy to PR something like the above. Be the same change for KafkaConsumerBundle too right? Any thing specific/different/enhanced etc you'd like to see in a PR?

[natalie-zamani]

I think we should include a property in the KafkaClientFactory class, since topics are common to both producers and consumers:

@NotNull
@JsonProperty
private Collection<String> topics = Collections.emptyList();

However, my original reasoning for keeping topics out of configuration is that Producers ultimately have to know what messages they should be sending to what topics, and that's not something the factory really handles. The only reason we take topics in the constructor for the KafkaProducerBundle is to register health checks currently. So it's still going to be up to the user to know what to do with those topics.

The issue with keeping topics in the contstructor, and adding another mechanism to get them is that I'm not sure what the right semantics are if topics were already provided in the constructor and then more/different ones added via that callback. The code could not call getKafkaTopics if there were any topics passed in at construction, or it could add to the existing.

My opinion here would be that we either merge the topics defined in configuration with the topics passed in the constructor, or we prefer the topics defined in the constructor over the configuration values (as a sort of override option).

I'm happy to PR something like the above. Be the same change for KafkaConsumerBundle too right? Any thing specific/different/enhanced etc you'd like to see in a PR?

So Consumers are a bit different, in that after the Consumer is actually built, I was leaving it up to users to actually call the consumer.subscribe(topics) method. I'm not totally sold on whether we should do that for a user or not, because there are risks of ConcurrentModificationException, and it may be that the user wishes to delay subscribing to topics for some reason. Maybe, by default, calling subscribe (but in a protected method that can be overridden by users).

[natalie-zamani]

Hey @michael-forman, is this still something you plan on taking a crack at?