dropwizard / dropwizard-kafka Goto Github PK

Please add Factory for org.apache.kafka.clients.admin.AdminClient

StringSerializer, for example has encoding configuration. How do you configure it?

https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java#L33

Rather than just one-line class configurations, I would recommend objects

valueSerializer:
  class: org.apache.kafka...StringSerializer
  config:
     value.serializer.encoding: "UTF-16"

Hi,

Do we have support for kafka schema registry?

Regards,
Kishore

Hi,

Relatively new to dropwizard (and new to Kafka!), but was looking at using this module as it seemed like the best way to get started.

In #22 this comment talks about KafkaProducerBundle not having a default constructor.

From looking at the doco and the source code, it would seem I need to pass in the topics to the constructor.

If I understand the lifecycle of bundles etc, doesn't this mean that the topics can't come from Configuration?

Am I missing something by wanting to retrieve the topics from configuration?

Please consider adding this project to the list of official Dropwizard modules at https://modules.dropwizard.io/.

https://github.com/dropwizard/modules.dropwizard.io/blob/8724c877edb55517a62f1662b9063e373a8d821a/_data/modules/official.yaml

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Base branch does not exist - skipping

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency io.dropwizard.modules:module-parent to v3 (release/2.1.x)
• Update dependency io.dropwizard.modules:module-parent to v4 (release/2.1.x)
• Update dependency io.dropwizard.modules:module-parent to v4 (release/3.0.x)

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

The 1.5.0 (at least) version of the library has the KafkaConsumerBundle constructor accepting a list of topics and a rebalance listener.

My understanding of the consumer rebalance listener is that it is a mechanism that allows the consumer to be warned of the rebalancing event so it has an opportunity to commit currently processing offsets.
That means that the rebalance listener needs a Consumer instance. Which it cannot have at that point since it is the bundle that is being created that can provide such instance.

I can circumvent by having a setter on the rebalancer listener for the consumer instance but that is confusing.

Kind of same remark for the topics argument, we don't have access to the configuration in the bootstrap() step so we cannot load the topics from configuration at that point.

Noticed releases are not added for 1.3.16,1.3.17, and 2.0 versions.

Now that AdminClient was added, I suggest adding a way that, during the lifecycle startup, then topics are created if they don't already exist.

Relevant methods:

• https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/admin/AdminClient.html#createTopics-java.util.Collection-
• https://kafka.apache.org/11/javadoc/org/apache/kafka/clients/admin/NewTopic.html

Configuration suggestion

topics:
  - name: foo
    partitions: 3
    replicationFactor: 3
  # topic with additional configuration
  - name: bar
    partitions: 1
    replicationFactor: 5
    config:
      cleanup.policy: compact

Additionally, would be nice to have a healthcheck that verifies topic existance + configuration details, but that can be future issue.

The KafkaConsumerBundle does not have any empty constructor, but one that takes a Collection<String> topics and a ConsumerRebalanceListener.

It does not seem like the topics actually does anything as I still, manually have to call subscribe / unsubscribe in my consumer. Is this correct?

Also, could someone please provide some documentation on what to implement in the ConsumerRebalanceListener. It has two methods, but is it OK to leave them empty? Do I have to do some manual commit of offsets here?

new ConsumerRebalanceListener() {
              @Override
              public void onPartitionsRevoked(final Collection<TopicPartition> collection) {
                // Intentionally empty
              }

              @Override
              public void onPartitionsAssigned(final Collection<TopicPartition> collection) {
                // Intentionally empty
              }
            }

I'm asking because I tried adding it to my project and the basic usage example in the README.md doesn't seem to work...

Also, what's with KafkaProducerBundler expecting the the dropwizard configuration type to inherit from javax.security.auth.login.Configuration rather than io.dropwizard.Configuration? Is this a bug?

The README is a good first step at docs, but its been suggested that there be code examples illustrating the usage of this library.

We should offer, preferably in the format of separate module(s), additional support for schema-based serialization formats, like Avro.

The integration test class DropwizardKafkaIT doesn't run as part of the build.

Steps:

• ./mvnw clean verify

Expected:
DropwizardKafkaIT is run

Actual:
DropwizardKafkaIT is not run

The current versioning scheme (pinning to the version of Dropwizard used in the project, and then having an incrementing counter: i.e., 1.3.12-2) seems to have some drawbacks that could be addressed by using semantic versioning. Namely, it's impossible to tell what a new version contains without looking at and comprehending release notes.

I think the main drawback with switching to SemVer would be the loss of obvious Dropwizard compatibility, which could be solved by having a compatibility matrix of sorts.

Discussion started here: https://groups.google.com/forum/?hl=en#!topic/dropwizard-dev/M97n4KXKyFA

With #9 it would be easy to add a cluster description healthcheck.

https://github.com/jeqo/dropwizard-modules/blob/master/kafka/src/main/java/io/github/jeqo/dropwizard/kafka/KafkaClientHealthCheck.java

At the moment, dropwizard-kafka does not support SASL plaintext authentication to Kafka server. The difference between SASL SSL and plaintext security is the absence of ssl.protocol configuration key, whose value is set to TLSv1.2, during setup.

Currently this project is plain consumer & producer. Can we get Kafka Streams configuration?

Maybe borrow some ideas from https://github.com/streamthoughts/azkarra-streams

Link streamthoughts/azkarra-streams#28

Hoping for some help from @joschi or any other @dropwizard-committers who's familiar with what the process is to get artifact publishing all set up!

Looks like there are releases after the current "latest" one marked .

https://github.com/dropwizard/dropwizard-kafka/releases