KafkaSparkStreamingSpec testsuite does not pass. about kafka-storm-starter

[miguno]

Hmm, I can't reproduce either of the errors you report, Huy.

Can you provide more details about your environment? For example, which OS, which Java version, whether you are running in a VM or not, whether you have modified the build, etc.

The first error is a timeout when Spark Streaming is trying to connect to ZooKeeper:

2014-10-09 19:43:09 org.apache.spark.streaming.scheduler.ReceiverTracker [ERROR] Deregistered receiver for stream 0: Error starting receiver 0 - org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 1000

You can try increasing the timeout in KafkaSparkStreamingSpec if you think that you're machine is too slow:

        val kafkaParams = Map[String, String](
          "zookeeper.connect" -> kafkaZkCluster.zookeeper.connectString,
          "group.id" -> sparkStreamingConsumerGroup,
          // CAUTION: Spark's use of auto.offset.reset is DIFFERENT from Kafka's behavior!
          // https://issues.apache.org/jira/browse/SPARK-2383
          "auto.offset.reset" -> "smallest", // must be compatible with when/how we are writing the input data to Kafka
          "zookeeper.connection.timeout.ms" -> "1000")

Here, change "zookeeper.connection.timeout.ms" from "1000" (1 sec) to "10000" (10 secs).

Alternatively, maybe ZK is not starting on your machine, and hence the timeout? You may want to increase logging in src/test/resources/logback-test.xml and see whether that helps you to pinpoint the problem:

  <root level="ERROR">     <!--  <<< change ERROR to DEBUG -->
    <appender-ref ref="A1"/>
  </root>

You may also want to read:

• SPARK-1949: Servlet 2.5 vs 3.0 conflict in SBT build
• http://wpcertification.blogspot.ch/2014/08/wordcount-program-built-using-apache.html (describes the "signer information does not match" problem and a workaround through dependency excludes)

I won't comment on the IDEA issue yet.

[huylv]

Hi Michael,

I've just tried to pull again from git and this time it works, both from command line and within IntelliJ. Strange thing is that yesterday I didn't even touch a file before running ./sbt test, just a git clone and then ./sbt test. I have no idea what's going on. The only difference is that I restarted Chrome before pulling from git (well, there's maybe nothing to do with Chrome, or maybe Chrome ate up all the memory and made Zookeeper go timeout?). I really appreciate your help, however.

By the way, I'm using Oracle Java 1.7.0_65 on Mavericks and everything's running directly from my machine.

Huy