im-pratham / kafka-perf-testing-k8s Goto Github PK
View Code? Open in Web Editor NEWK8s jobs to benchmark Kafka cluster on Kubernetes
License: Apache License 2.0
K8s jobs to benchmark Kafka cluster on Kubernetes
License: Apache License 2.0
Hi. First of all, thanks for the contribution. I am trying to conduct a performance test on the kafka cluster based on your contribution.
helm install confluentinc . -n kafka
kafka-client
it has been necessary to modify it:confluentinc/cp-server:7.2.2.amd64
PLAINTEXT://confluentinc-cp-kafka:9092
apiVersion: v1
kind: Pod
metadata:
name: kafka-client
labels:
app: kafka-examples
purpose: examples
type: client
spec:
containers:
- name: kafka-client
image: confluentinc/cp-server:7.2.2.amd64
command:
- sh
- -c
- "exec tail -f /dev/null"
env:
- name: BOOTSTRAP_SERVERS
value: PLAINTEXT://confluentinc-cp-kafka:9092
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: '1'
memory: 1Gi
KHOST=$(kubectl get svc -n kafka -o jsonpath="{.items[1].metadata.name}")
echo $KHOST
confluentinc-cp-kafka
KPORT=$(kubectl get svc -n kafka -o jsonpath="{.items[1].spec.ports[0].port}")
echo $KPORT
9092
export KAFKA=$KHOST:$KPORT
echo $KAFKA
confluentinc-cp-kafka:9092
cp-server:7.2.2.amd64
) does not contain the configuration file (/etc/kafka/config/client.properties
), it does contain the file (/etc/kafka/server.properties
), so the topic could be deployed.kubectl exec -it kafka-client -n kafka -- kafka-topics --bootstrap-server $KAFKA --create --topic ssl-perf-test --partitions 6 --replication-factor 3 --config retention.ms=1800000 --config min.insync.replicas=2 --command-config /etc/kafka/server.properties
single-producer.yaml
due to errors with the original image: containers:
- name: kafka-client
image: confluentinc/cp-server:7.2.2.amd64
env:
- name: BOOTSTRAP_SERVERS
value: PLAINTEXT://confluentinc-cp-kafka:9092
- name: KAFKA_ZOOKEEPER_CONNECT
value: "confluentinc-cp-zookeeper:2181"
- name: KAFKA_ADVERTISED_LISTENERS
value: PLAINTEXT://confluentinc-cp-zookeeper:5556
kubectl apply -f single-producer.yaml -n kafka
configmap/kafka-single-producer-perf-script unchanged
job.batch/kafka-single-producer-client created
NAME READY STATUS RESTARTS AGE
pod/kafka-single-producer-client--1-lz9kt 1/1 Running 0 84s
NAME COMPLETIONS DURATION AGE
job.batch/kafka-single-producer-client 0/1 88s 90s
At the end, I get this exception: org.apache.kafka.common.errors.TimeoutException: Call has timed out. Callback: fetchMetadata
.
I leave here the link to the huge log obtained.
Any suggestion?
Hi again. I am trying to deploy the same kafka cluster (already deployed in minikube) on k8s.
As you can see the pods do not deploy successfully.
Error logs for kafka-cp-control-center:
[main] WARN org.apache.kafka.clients.ClientUtils - Couldn't resolve server PLAINTEXT://kafka-cp-kafka-headless:9092 from bootstrap.servers as DNS resolution failed for kafka-cp-kafka-headless
[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:535)
at org.apache.kafka.clients.admin.Admin.create(Admin.java:75)
at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)
at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)
at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:489)
... 4 more
Error logs for kafka-cp-kafka-connect:
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
Error logs for kafka-cp-kafka-rest:
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 27 more
Error logs for kafka-cp-ksql-server:
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
Error logs for kafka-cp-ksql-server:
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
... 26 more
Have you deployed a kafka cluster on k8s before?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.