azure / toketi-kafka-connect-iothub Goto Github PK

connector.class=com.microsoft.azure.iot.kafka.connect.source.IotHubSourceConnector

needs to be replaced with

connector.class=com.microsoft.azure.iot.kafka.connect.IotHubSourceConnector

in the https://github.com/Azure/toketi-kafka-connect-iothub/blob/master/connect-iothub-source.properties file

I've been working with this connector for a few months now. Recently we've noticed in our staging environment that some of the connector tasks were entering a failed state.

Here are a few failed connector tasks that I've seen. This first one says Exception - com.microsoft.azure.servicebus.ServiceBusException: The message container is being closed (428).

{
  "name": "uploadnotifications_in",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-2.connect:28082"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "connect-0.connect:28082"
    },
    {
      "state": "FAILED",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Error while polling for data at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask.poll(IotHubSourceTask.scala:37) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:163) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kafka.connect.errors.ConnectException: Error while getting SourceRecords for partition 1. Exception - com.microsoft.azure.servicebus.ServiceBusException: The message container is being closed (428). TrackingId:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_B24, SystemTracker:NoSystemTracker, Timestamp:12/18/2017 10:44:58 AM, errorContext[NS: xxxxxxxxxx.servicebus.windows.net, PATH: uploadnotifications/ConsumerGroups/kafka-cg/Partitions/1, REFERENCE_ID: xxxxxx_xxx_xxx_xxxxxxxxxxxxx, PREFETCH_COUNT: 999, LINK_CREDIT: 999, PREFETCH_Q_LEN: 0] Stack trace - () at com.microsoft.azure.iot.kafka.connect.source.IotHubPartitionSource.getRecords(IotHubPartitionSource.scala:51) at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask$$anonfun$poll$1.apply(IotHubSourceTask.scala:30) at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask$$anonfun$poll$1.apply(IotHubSourceTask.scala:28) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask.poll(IotHubSourceTask.scala:28) ... 8 more Caused by: com.microsoft.azure.servicebus.ServiceBusException: The message container is being closed (428). TrackingId:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_B24, SystemTracker:NoSystemTracker, Timestamp:12/18/2017 10:44:58 AM, errorContext[NS: xxxxxxxxxx.servicebus.windows.net, PATH: uploadnotifications/ConsumerGroups/kafka-cg/Partitions/1, REFERENCE_ID: xxxxxx_xxx_xxx_xxxxxxxxxxxxx, PREFETCH_COUNT: 999, LINK_CREDIT: 999, PREFETCH_Q_LEN: 0] at com.microsoft.azure.servicebus.ExceptionUtil.toException(ExceptionUtil.java:60) at com.microsoft.azure.servicebus.MessageReceiver.onClose(MessageReceiver.java:587) at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:75) at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:46) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:309) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:276) at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:375) ... 1 more ",
      "id": 1,
      "worker_id": "connect-2.connect:28082"
    }
  ]
}

This second one is from a connector that is connected to a different EventHub resource. It has encountered a different error: Exception - com.microsoft.azure.servicebus.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The service was unable to process the request; please retry the operation.

{
  "name": "uploadnotifications_in",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-2.connect:28082"
  },
  "tasks": [
    {
      "state": "RUNNING",
      "id": 0,
      "worker_id": "connect-0.connect:28082"
    },
    {
      "state": "FAILED",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Error while polling for data at com.microsoft.azure.iot.kafka.connect.IotHubSourceTask.poll(IotHubSourceTask.scala:37) at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:163) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:146) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:190) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kafka.connect.errors.ConnectException: Error while getting SourceRecords for partition 13. Exception - com.microsoft.azure.servicebus.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_G4, SystemTracker:gateway6, Timestamp:12/18/2017 6:09:02 AM, errorContext[NS: xxxxxxxxxxxxxxx.servicebus.windows.net, PATH: uploadnotifications/ConsumerGroups/kafka-cg/Partitions/13, REFERENCE_ID: xxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_G4, PREFETCH_COUNT: 999, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0, R_TYPE: NON_EPOCH] Stack trace - () at com.microsoft.azure.iot.kafka.connect.IotHubPartitionSource.getRecords(IotHubPartitionSource.scala:51) at com.microsoft.azure.iot.kafka.connect.IotHubSourceTask$$anonfun$poll$1.apply(IotHubSourceTask.scala:30) at com.microsoft.azure.iot.kafka.connect.IotHubSourceTask$$anonfun$poll$1.apply(IotHubSourceTask.scala:28) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) at com.microsoft.azure.iot.kafka.connect.IotHubSourceTask.poll(IotHubSourceTask.scala:28) ... 8 more Caused by: com.microsoft.azure.servicebus.ServiceBusException: com.microsoft.azure.servicebus.amqp.AmqpException: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_G4, SystemTracker:gateway6, Timestamp:12/18/2017 6:09:02 AM, errorContext[NS: xxxxxxxxxxxxxxx.servicebus.windows.net, PATH: uploadnotifications/ConsumerGroups/kafka-cg/Partitions/13, REFERENCE_ID: xxxxxx_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_G4, PREFETCH_COUNT: 999, LINK_CREDIT: 0, PREFETCH_Q_LEN: 0, R_TYPE: NON_EPOCH] at com.microsoft.azure.servicebus.ExceptionUtil.toException(ExceptionUtil.java:58) at com.microsoft.azure.servicebus.MessageReceiver.onError(MessageReceiver.java:393) at com.microsoft.azure.servicebus.MessageReceiver.onClose(MessageReceiver.java:646) at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.processOnClose(BaseLinkHandler.java:83) at com.microsoft.azure.servicebus.amqp.BaseLinkHandler.onLinkRemoteClose(BaseLinkHandler.java:52) at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:176) at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:309) at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:276) at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:340) ... 1 more Caused by: com.microsoft.azure.servicebus.amqp.AmqpException: The service was unable to process the request; please retry the operation. For more information on exception types and proper exception handling, please refer to http://go.microsoft.com/fwlink/?LinkId=761101 TrackingId:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx_G4, SystemTracker:gateway6, Timestamp:12/18/2017 6:09:02 AM ... 11 more ",
      "id": 1,
      "worker_id": "connect-2.connect:28082"
    }
  ]
}

We think the first failure above is related to this issue where the Azure microservice that is hosting the eventhub needs to restart, thus disconnecting the receivers. Is it possible to try to handle these disconnects from the eventhub in this source connector? Or is it better to handle this in the azure-event-hubs library?

Our current approach is to monitor the connector status endpoints with a separate system and restart the connectors and tasks when necessary but I'm wondering if there is a better way.

For the record, we're connecting directly to a separate eventhub, not the actual IoTHub. We're doing this because we're routing some specific device message types to different custom endpoints in IoTHub, and those endpoints are backed by eventhubs. It's these eventhubs that we're connecting to with Kafka-connect. I'm not sure if that matters at all.

Upon following the instructions specified in the Kafka Connect plugin for Azure IoTHub source i.e., prepared the properties file and placed in the config directory and placed the JAR file of azure IoT hub in the lib directory.

When both the zookeeper and Kafka servers are running, I ran the following command:
bin/connect-standalone.sh config/connect-standalone.properties config/connect-iothub-source.properties

The command when ran is giving following error messages... Not sure if Microsoft is supporting this feature of Kafka Connect for IoTHub...

ERROR Plugin class loader for connector: 'com.microsoft.azure.iot.kafka.connect.IotHubSourceConnector' was not found. Returning: org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader@423e4cbb

ERROR WorkerSourceTask{id=AzureIotHubConnector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)

ERROR WorkerSourceTask{id=AzureIotHubConnector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:180)

[2020-04-14 16:05:31,694] ERROR WorkerSourceTask{id=AzureIotHubConnector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:179)
com.microsoft.azure.servicebus.CommunicationException: java.nio.channels.UnresolvedAddressException. This is usually caused by incorrect hostname or network configuration. Please check to see if namespace information is correct. TrackingId: 16cca8fa-bb21-4c4c-84d7-6695f20de2ac, at: 2020-04-14T16:05:31.692+05:30[Asia/Kolkata]
at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:371)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:101)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
at org.apache.qpid.proton.reactor.impl.IOHandler.handleBound(IOHandler.java:155)
at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:372)
at org.apache.qpid.proton.engine.BaseHandler.onConnectionBound(BaseHandler.java:58)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:131)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:309)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:277)
at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:340)
... 1 more

Can someone help me in understanding what exactly going wrong...?

Looks like EventHubReceiver simply wraps IoT Hub Messages in an IoTMessage instance which in turn wraps the underlying EventHub Message in a scala Map and sends it to Kafka. This adds quite a bit of complexity on the Kafka consumer side when trying to deserialize the original IoT Hub message.

Hey Varun,

I followed exact steps but there was one issue, i have 2 partitions on my iothub and source connector reads from just 1 partition not from other one.
I have tried with 1 and 2 partitions in kafka topic.

Please help sort it out.

Thanks

C:\kafka_2.11-1.0.1>.\bin\windows\connect-standalone.bat .\config\connect-standalone.properties .\config\connect-iothub-source.properties >> connecterror.txt
[2018-03-27 09:17:40,710] WARN could not get type for name org.osgi.framework.BundleListener from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.osgi.framework.BundleListener
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.osgi.framework.BundleListener
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,724] WARN could not get type for name org.apache.kafka.common.utils.MockTime from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.common.utils.MockTime
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.MockTime
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,745] WARN could not get type for name org.scalatest.junit.JUnitSuite from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.junit.JUnitSuite
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.scalatest.junit.JUnitSuite
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,798] WARN could not get type for name org.easymock.IAnswer from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.easymock.IAnswer
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.easymock.IAnswer
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,806] WARN could not get type for name org.jboss.netty.channel.MessageEvent from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.jboss.netty.channel.MessageEvent
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.MessageEvent
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,845] WARN could not get type for name org.jboss.netty.channel.SimpleChannelHandler from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.jboss.netty.channel.SimpleChannelHandler
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.jboss.netty.channel.SimpleChannelHandler
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,854] WARN could not get type for name javax.mail.Authenticator from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name javax.mail.Authenticator
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: javax.mail.Authenticator
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,887] WARN could not get type for name org.apache.kafka.test.MockMetricsReporter from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.apache.kafka.test.MockMetricsReporter
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.test.MockMetricsReporter
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,926] WARN could not get type for name javax.jms.MessageListener from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name javax.jms.MessageListener
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: javax.jms.MessageListener
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,949] WARN could not get type for name org.eclipse.jetty.jmx.ObjectMBean from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.eclipse.jetty.jmx.ObjectMBean
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.jmx.ObjectMBean
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:40,998] WARN could not get type for name org.osgi.framework.BundleActivator from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.osgi.framework.BundleActivator
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.osgi.framework.BundleActivator
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:41,027] WARN could not get type for name org.scalatest.Assertions from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.scalatest.Assertions
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.scalatest.Assertions
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:41,058] WARN could not get type for name org.osgi.framework.SynchronousBundleListener from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name org.osgi.framework.SynchronousBundleListener
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: org.osgi.framework.SynchronousBundleListener
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
[2018-03-27 09:17:41,087] WARN could not get type for name jline.Completor from any class loader (org.reflections.Reflections)
org.reflections.ReflectionsException: could not get type for name jline.Completor
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
at org.reflections.Reflections.(Reflections.java:126)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:263)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:206)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initPluginLoader(DelegatingClassLoader.java:154)
at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:147)
at org.apache.kafka.connect.runtime.isolation.Plugins.(Plugins.java:47)
at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:75)
Caused by: java.lang.ClassNotFoundException: jline.Completor
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
... 8 more
Mar 27, 2018 9:17:42 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: The (sub)resource method createConnector in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectors in org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource contains empty path annotation.
WARNING: The (sub)resource method listConnectorPlugins in org.apache.kafka.connect.runtime.rest.resources.ConnectorPluginsResource contains empty path annotation.
WARNING: The (sub)resource method serverInfo in org.apache.kafka.connect.runtime.rest.resources.RootResource contains empty path annotation.

So when some one gets access to the connector properties my customer doesn't want to see secret key. they want the secret key to store in external key store like troll bridge or something. This is a big deal with their CISO organization.

I have created a jar by using the command sbt assembly. Java 1.8 and scala 2.11. The Jar was working well on my Mac but when i transported my jar to ubuntu i ran into this error while creating a connector using Kafka connect Rest API.

java.lang.IncompatibleClassChangeError: Method 'com.microsoft.azure.eventhubs.EventHubClient com.microsoft.azure.eventhubs.EventHubClient.createSync(java.lang.String, java.util.concurrent.Executor)' must be InterfaceMethodref constant
at com.microsoft.azure.iot.kafka.connect.source.EventHubReceiver.(EventHubReceiver.scala:19)
at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask.getDataReceiver(IotHubSourceTask.scala:96)
at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask$$anonfun$start$2.apply(IotHubSourceTask.scala:86)
at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask$$anonfun$start$2.apply(IotHubSourceTask.scala:68)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at scala.collection.immutable.Map$Map2.foreach(Map.scala:137)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at com.microsoft.azure.iot.kafka.connect.source.IotHubSourceTask.start(IotHubSourceTask.scala:68)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:213)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

This release jar file has broken structure for Source connector. Inside jar, the path com/microsoft/azure/iot/kafka/connect contains folder sink but not source which causes Kafka Connect to not able to load the plugin. It cannot find class com.microsoft.azure.iot.kafka.connect.source.IotHubSourceConnector .. Rebuilding the package from source works. Please rebuild it and upload the correct release artifact.

Thanks

We want to connect our Azure IoT Hub with a Kafka-based Azure HD Insight Cluster and are getting the same error as documented in: Azure/azure-iot-sdk-java#491 and Azure/azure-service-bus-java#332 (Log Error below)

Our setup:

• HD Insight Cluster
• IoT Hub
• Docker Image based on Confluent cp-kafka-connect with kafka-connect-iothub:0.6

FROM confluentinc/cp-kafka-connect
RUN confluent-hub install microsoft/kafka-connect-iothub:0.6 --no-prompt
ADD connect-iothub-source.properties /usr/share/confluent-hub-components/microsoft-kafka-connect-iothub/etc/connect-iothub-source.properties

We configured the kafka-connect-iothub connector via the REST API with curl. On startup, the error occurs.

ERROR WorkerSourceTask{id=iothub-source-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
com.microsoft.azure.servicebus.ServiceBusException: org.apache.qpid.proton.engine.TransportException: None of [TLS_DH_anon_WITH_AES_128_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA] anonymous cipher suites are within the supported list [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV], TrackingId: 9bc973b1-3eca-425c-928f-7205170165fa, at: 2019-10-07T13:26:59.749Z[Etc/UTC]
        at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:365)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.qpid.proton.engine.TransportException: org.apache.qpid.proton.engine.TransportException: None of [TLS_DH_anon_WITH_AES_128_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA] anonymous cipher suites are within the supported list [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
        at org.apache.qpid.proton.engine.impl.ssl.SslImpl$UnsecureClientAwareTransportWrapper.pending(SslImpl.java:166)
        at org.apache.qpid.proton.engine.impl.TransportImpl.pending(TransportImpl.java:1465)
        at org.apache.qpid.proton.engine.impl.TransportImpl.isClosed(TransportImpl.java:1582)
        at org.apache.qpid.proton.reactor.impl.IOHandler.capacity(IOHandler.java:174)
        at org.apache.qpid.proton.reactor.impl.IOHandler.update(IOHandler.java:204)
        at org.apache.qpid.proton.reactor.impl.IOHandler.selectableTransport(IOHandler.java:329)
        at org.apache.qpid.proton.reactor.impl.IOHandler.handleBound(IOHandler.java:166)
        at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:372)
        at org.apache.qpid.proton.engine.BaseHandler.onConnectionBound(BaseHandler.java:58)
        at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:131)
        at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:309)
        at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:277)
        at com.microsoft.azure.servicebus.MessagingFactory$RunReactor.run(MessagingFactory.java:340)
        ... 1 more
Caused by: org.apache.qpid.proton.engine.TransportException: None of [TLS_DH_anon_WITH_AES_128_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA] anonymous cipher suites are within the supported list [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
        at org.apache.qpid.proton.engine.impl.ssl.SslEngineFacadeFactory.buildEnabledSuitesIncludingAnonymous(SslEngineFacadeFactory.java:397)
        at org.apache.qpid.proton.engine.impl.ssl.SslEngineFacadeFactory.addAnonymousCipherSuites(SslEngineFacadeFactory.java:376)
        at org.apache.qpid.proton.engine.impl.ssl.SslEngineFacadeFactory.createAndInitialiseSslEngine(SslEngineFacadeFactory.java:212)
        at org.apache.qpid.proton.engine.impl.ssl.SslEngineFacadeFactory.createProtonSslEngine(SslEngineFacadeFactory.java:184)
        at org.apache.qpid.proton.engine.impl.ssl.SslDomainImpl.createSslEngine(SslDomainImpl.java:132)
        at org.apache.qpid.proton.engine.impl.ssl.SslImpl$UnsecureClientAwareTransportWrapper.initTransportWrapperOnFirstIO(SslImpl.java:231)
        at org.apache.qpid.proton.engine.impl.ssl.SslImpl$UnsecureClientAwareTransportWrapper.capacity(SslImpl.java:108)
        at org.apache.qpid.proton.engine.impl.TransportImpl.capacity(TransportImpl.java:1428)
        at org.apache.qpid.proton.reactor.impl.IOHandler.capacity(IOHandler.java:172)
        ... 11 more
[2019-10-07 13:26:59,759] ERROR WorkerSourceTask{id=iothub-source-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2019-10-07 13:26:59,760] INFO Stopping IotHubSourceTask (com.microsoft.azure.iot.kafka.connect.IotHubSourceTask)

The Java version is:

docker exec kafka-connect java -version
openjdk version "1.8.0_212"
OpenJDK Runtime Environment (Zulu 8.38.0.13-CA-linux64) (build 1.8.0_212-b04)
OpenJDK 64-Bit Server VM (Zulu 8.38.0.13-CA-linux64) (build 25.212-b04, mixed mode)

Hi Team,

Does SSl Encryption Supported in IOT Hub Source Connector. If yes can you please provide example of propertie file

Regards,
Hari

We have cloudera cluster with 3 kafka brokers in it. In this case, do we need to run standalone script from all the 3 brokers or only the edge node?

can you also provide steps to run it in distributed mode too? also we are in cluster with JDK 1.7, is there any requirement for this connector to be on JDK 1.8?

Could you please release newer versions? Or maybe at lease tag them?
Thanks

Could "CompatibleNamespace" property changed to align with the actual value form the portal ... say something like "CompatibleEndpointHost" given that "Endpoint" is used in the Azure Portal for that value.

We tried to use the Toketi Kafka Connector​ (The Source Connector) but failed to implement it.

See the repro steps to be able to reproduce the error we get.

Requirement

• ​Having a running IOT Hub architecture (IOT Edge in our case)
• Having an Azure Kafka HD Insight running

Uploading Binaries on kafka

​Downloading binaries / conf file, locally
Editing the conf file to match the IOT Hub conf

​connector.class=com.microsoft.azure.iot.kafka.connect.source.IotHubSourceConnector
name=AzureIotHubConnector
tasks.max=1
Kafka.Topic=IotTopic
IotHub.EventHubCompatibleName=iothubseb
IotHub.EventHubCompatibleEndpoint=sb://iothub-ns-iothubseb-403463-ff108f0690.servicebus.windows.net/
IotHub.AccessKeyName=service
IotHub.AccessKeyValue=XahyUfVfU1IYJON1dtqW7qG67Oe9HiowX3dcnTrxdk8=
IotHub.ConsumerGroup=$Default
IotHub.Partitions=4
IotHub.StartTime=2018-01-28T00:00:00Z
IotHub.Offsets=
BatchSize=100
ReceiveTimeout=60

Then uploading IOT Hub jar & config To Kafka

scp kafka-connect-iothub-assembly_2.11-0.6.jar [email protected]:/home
/sshuser/
scp connect-iothub-source.properties [email protected]:/home/sshuser/​

Copying binaries in the right folder

Connect with SSH on Kafka then copying the files to correct directories:

cp kafka-connect-iothub-assembly_2.11-0.6.jar /usr/hdp/current/kafka-broker/libs/
cp connect-iothub-source.properties /usr/hdp/current/kafka-broker/config

Go to Kafka config directory

cd /usr/hdp/current/kafka-broker

Launch the command

​ bin/connect-standalone.sh config/connect-standalone.properties config/connect-iothub-source.properties ​

Error raised :

could not create Vfs.Dir from url, no matching UrlType was found [file:/usr/hdp/current/kafka-broker/bin/../libs/rolling_upgrade.sh]
java.io.FileNotFoundException: /usr/hdp/current/kafka-broker/bin/../libs/zookeeper.jar
java.io.FileNotFoundException: /usr/hdp/current/kafka-broker/bin/../libs/ojdbc6.jar
org.apache.kafka.connect.errors.ConnectException: Failed to start connector: AzureIotHubConnector
ERROR Failed to start connector AzureIotHubConnector (org.apache.kafka.connect.runtime.Worker:167)
org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.microsoft.azure.iot.kafka.connect.source.IotHubSourceConnector, available connectors are: org.apache.kafka.connect.source.SourceConnector, org.apache.kafka.connect.tools.SchemaSourceConnector, org.apache.kafka.connect.sink.SinkConnector, org.apache.kafka.connect.tools.MockConnector, org.apache.kafka.connect.tools.MockSinkConnector, com.microsoft.azure.iot.kafka.connect.IotHubSourceConnector, org.apache.kafka.connect.tools.VerifiableSinkConnector, com.microsoft.azure.iot.kafka.connect.sink.IotHubSinkConnector, org.apache.kafka.connect.file.FileStreamSinkConnector, org.apache.kafka.connect.tools.VerifiableSourceConnector, org.apache.kafka.connect.file.FileStreamSourceConnector, org.apache.kafka.connect.tools.MockSourceConnector​