ekoutanov / effectivekafka

Effective Kafka

Induces failure of confluent-kafka-go

Relates to https://github.com/confluentinc/confluent-kafka-go/issues/455.

To run:

make && bin/kafkaload

It may take a while...

effectivekafka's Issues

SSL Problem

The ssl key generator doesn't work properly:

Failed to load SSL keystore /home/mahdi/Documents/kafka/kafka_2.13-2.8.0/config/server.keystore.jks of type JKS
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.(DefaultSslEngineFactory.java:349)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:138)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:107)
at kafka.network.Processor.(SocketServer.scala:853)
at kafka.network.SocketServer.newProcessor(SocketServer.scala:442)
at kafka.network.SocketServer.$anonfun$addDataPlaneProcessors$1(SocketServer.scala:299)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:190)
at kafka.network.SocketServer.addDataPlaneProcessors(SocketServer.scala:297)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:262)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:259)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:259)
at kafka.network.SocketServer.startup(SocketServer.scala:131)
at kafka.server.KafkaServer.startup(KafkaServer.scala:285)
at kafka.Kafka$.main(Kafka.scala:109)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.io.IOException: Invalid keystore format
at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:667)
at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:222)
at java.base/java.security.KeyStore.load(KeyStore.java:1479)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374)
... 23 more
[2021-08-01 21:39:31,805] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
[2021-08-01 21:39:31,806] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Stopping socket server request processors (kafka.network.SocketServer)
[2021-08-01 21:39:31,809] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Stopped socket server request processors (kafka.network.SocketServer)
[2021-08-01 21:39:31,812] INFO Shutting down. (kafka.log.LogManager)
[2021-08-01 21:39:31,845] INFO Shutdown complete. (kafka.log.LogManager)
[2021-08-01 21:39:31,846] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-08-01 21:39:31,846] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-08-01 21:39:31,846] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2021-08-01 21:39:31,846] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-08-01 21:39:31,951] INFO Session: 0x1000324c18a0000 closed (org.apache.zookeeper.ZooKeeper)
[2021-08-01 21:39:31,951] INFO EventThread shut down for session: 0x1000324c18a0000 (org.apache.zookeeper.ClientCnxn)
[2021-08-01 21:39:31,952] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-08-01 21:39:31,952] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:32,291] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:32,291] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:32,292] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:33,292] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:33,292] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:33,292] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,292] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,292] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,293] INFO [ThrottledChannelReaper-ControllerMutation]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,293] INFO [ThrottledChannelReaper-ControllerMutation]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,293] INFO [ThrottledChannelReaper-ControllerMutation]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2021-08-01 21:39:34,294] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Shutting down socket server (kafka.network.SocketServer)
[2021-08-01 21:39:34,319] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Shutdown completed (kafka.network.SocketServer)
[2021-08-01 21:39:34,320] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2021-08-01 21:39:34,320] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2021-08-01 21:39:34,320] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2021-08-01 21:39:34,322] INFO Broker and topic stats closed (kafka.server.BrokerTopicStats)
[2021-08-01 21:39:34,324] INFO App info kafka.server for 0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2021-08-01 21:39:34,325] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
[2021-08-01 21:39:34,325] ERROR Exiting Kafka. (kafka.Kafka$)
[2021-08-01 21:39:34,325] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)

What has been used to create the beautiful diagrams within the book?

First of all I want to say a big thank you for the incredible book!
While reading it I noticed how great the quality of the Diagrams within the book are.
I especially like this one:

Could you perhaps share what tools you used for the diagrams?

Is there any errdata about the book?

I am going through page 150 (PDF), chapter 9, and the last paragraph for the section "Dynamic Configuration" says:

Now that we’re done with our examples, we can revert the configuration to its original state by
deleting the cluster-wide entry:

kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type brokers --entity-default \
--describe num.io.threads

I was expecting to see something in the order:

kafka-configs.sh --bootstrap-server localhost:9092 \
--entity-type brokers --entity-default \
--alter --delete-config num.io.threads

but I am unsure. So I just want to double-check if the book is correct and if the parameter --describe actually applies changes to the cluster.

And a question more, is there any errdata for the book where one can check whether something is an error or not? I could not find anything either here or in leanpub.

Thanks!