Incorrect broker's address extraction about spring-integration-kafka

[garyrussell]

There is no manipulation of the String value for the broker list; it is passed into Kafka as a property unchanged here. Perhaps you have some other version of your config with the comma on your classpath?

[rssdev10]

Thanks, and sorry, I'm not familiar with Spring Integration. I tried to use your examples and description from https://github.com/spring-projects/spring-integration-kafka. Like this

<int-kafka:message-driven-channel-adapter
            id="adapter"
            channel="output"
            connection-factory="connectionFactory"
            key-decoder="decoder"
            payload-decoder="decoder"
            offset-manager="offsetManager"
            max-fetch="100"
            topics="${kafka.test.topic}"/>

How to configure the broker list directly? https://github.com/rssdev10/flink-spring-integration/blob/master/drpc-client/src/main/resources/flink-integration-context.xml

BTW: what is the topics string format?

And regarding Kafka I nothing changed in they typical config.

[artembilan]

Try to use BrokerAddressListConfiguration instead of ZookeeperConfiguration.
Also it is really isn't clear why your current solution fail with the strange resolution to the localhost,:9092.
The code to consider ( ZookeeperConfiguration):

    zkClient = new ZkClient(this.zookeeperServers, this.sessionTimeout, this.connectionTimeout,
                    ZKStringSerializer$.MODULE$);
            Seq<Broker> allBrokersInCluster = ZkUtils$.MODULE$.getAllBrokersInCluster(zkClient);
            FastList<Broker> brokers = FastList.newList(JavaConversions.asJavaCollection(allBrokersInCluster));
            return brokers.collect(brokerToBrokerAddressFunction);

and DefaultConnectionFactory:

String brokerAddressesAsString = ListIterate
                    .collect(this.configuration.getBrokerAddresses(), Functions.getToString()).makeString(",");

[rssdev10]

@artembilan thanks, BrokerAddressConfiguration works.

    <bean id="connectionFactory"
        class="org.springframework.integration.kafka.core.DefaultConnectionFactory">
        <!-- <constructor-arg ref="zookeeperConfiguration" />  -->
        <constructor-arg ref="brokerConfig" />
    </bean>

    <bean id="brokerConfig"
        class="org.springframework.integration.kafka.core.BrokerAddressListConfiguration">
        <constructor-arg value="#{T(org.springframework.integration.kafka.core.BrokerAddress).fromAddress('localhost:9092')}" />
    </bean>

But is it possible to configure one without such terrible SpEL?

[artembilan]

Just make BrokerAddress as inner bean!

 <bean id="brokerConfig"
        class="org.springframework.integration.kafka.core.BrokerAddressListConfiguration">
        <constructor-arg>
             <bean class="org.springframework.integration.kafka.core.BrokerAddress" factory-method="fromAddress">
               <constructor-arg value="localhost:9092"/>
            </bean>
        </constructor-arg>
    </bean>

But SpEL should be shorter.
From other side it would be great to understand who breaks localhost:9092 to the localhost,:9092 for you in case of ZookeeperConfiguration...

Would you mind to debug your application with break points on the Spring Integration Kafka code, e.g. ZookeeperConfiguration.doGetBrokerAddresses() will be a good candidate?

[rssdev10]

[artembilan]

M-m-m. Any comments, please. I'm not familiar with Eclipse:
The screenshot from IDEA:

[artembilan]

According to your screen it really shows that Zookeeper returns localhost, as a host for your broker.
Not sure where that is broken, but right now it is out of our hands.
We can't control it on Spring Integration Kafka side.

[rssdev10]

Eclipse is not so hard :) . Left upper part contains a current state of variables and "localhost," is shown.

I guess the problem is somewhere in ZkUtils / getBrokerInfo(zkClient, _).

  def getAllBrokersInCluster(zkClient: ZkClient): Seq[Broker] = {
    val brokerIds = ZkUtils.getChildrenParentMayNotExist(zkClient, ZkUtils.BrokerIdsPath).sorted
    brokerIds.map(_.toInt).map(getBrokerInfo(zkClient, _)).filter(_.isDefined).map(_.get)
  }

Kafka for me is new product too. I can't comment anything.

[rssdev10]

I found the reason. /etc/hosts contained a comma as a separator....

127.0.0.1       localhost, linux-...

When I fixed it and restarted Kafka the application was started correctly.
Thanks for the advises. And please add error info in doGetBrokerAddresses() to avoid anonymous java.lang.NullPointerException.

[artembilan]

Thank you for the final solution!

Yes, we definitely should fix it.
Doing that now...

[artembilan]

Well, I'm not sure that we can do here so much:

java.lang.NullPointerException
    at scala.Predef$.Integer2int(Predef.scala:392)
    at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:103)
    at kafka.client.ClientUtils$$anonfun$parseBrokerList$1.apply(ClientUtils.scala:102)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.AbstractTraversable.map(Traversable.scala:105)
    at kafka.client.ClientUtils$.parseBrokerList(ClientUtils.scala:102)
    at org.springframework.integration.kafka.core.DefaultConnectionFactory.refreshMetadata(DefaultConnectionFactory.java:175)

Looks like Kafka should take care about the null for the port.

Anyway, I'll throw some IllegalState for the ClientUtils$.parseBrokerList with the appropriate brokerAddressesAsString to see the issue in the logs.

[artembilan]

See #98