confluentinc / confluent-kubernetes-examples

Example scenario workflows for Confluent for Kubernetes

Confluent for Kubernetes Scenario Examples

This GitHub repository accompanies the official Confluent for Kubernetes documentation.

This repository contains scenario workflows to deploy and manage Confluent on Kubernetes for various use cases.

Prerequisites

The following prerequisites are assumed for each scenario workflow:

• A Kubernetes cluster - any CNCF conformant version
• Helm 3 installed on your local machine
• Kubectl installed on your local machine
• A namespace created in the Kubernetes cluster - confluent
• Kubectl configured to target the confluent namespace:

  kubectl config set-context --current --namespace=confluent
  

• This repo cloned to your workstation:

  git clone [email protected]:confluentinc/confluent-kubernetes-examples.git
  

Next Steps

You can browse through our curated example scenarios and try out the ones that matter to you.

If you are looking for inspiration, below are some of the popular scenarios.

Quickstart

In this workflow scenario, you'll set up a simple non-secure (no authn, authz or encryption) Confluent Platform, consisting of all components. You can also setup up a simple KRaft based deployment as well.

The goal for this scenario is for you to:

Quickly set up the complete Confluent Platform on the Kubernetes. Configure a producer to generate sample data. Head to CFK Quickstart.

We are adopting KRaft in place of Zookeeper. Give it a quick try here.

Security

We have curated a comprehensive list of examples of commonly used security configurations. Some of the notable ones are:

• SASL/Plain with LDAP for authentication & authorization, TLS for encryption using CFK auto-generated component certificates. You'll need to provide a certificate authority certificate for CFK to auto-generate the component certificates. Example.
• TLS encryption using user provided certificates, mTLS authentication, Confluent RBAC authorization Example.

For more security use cases, click here.

Networking

External Access Load Balancer Deployment in CFK

This example provides a way to enable external access to Confluent Platform components using load balancers, enhancing the accessibility and flexibility of your Confluent Platform deployment. It demonstrates

• How to configure load balancers for external access to Confluent Platform components in a Kubernetes environment.
• Setting the domain name of your Kubernetes cluster and configuring session affinity.

Advanced Configurations

Blueprints

Blueprint is a configuration template for Confluent Platform deployments using Confluent for Kubernetes. You can use it to enforce infrastructure and deployment standards within your organization. To try out Blueprints, click here.

Advanced Configuration with Pod Overlay

This example provides a way to leverage additional Kubernetes features not currently supported in the Confluent for Kubernetes (CFK) API, enhancing the flexibility and control over your Confluent Platform deployments.

• The Pod Overlay feature allows you to use ConfigMap and configure a StatefulSet PodTemplate for Confluent Platform components like Zookeeper, Kafka, Connect, Schema Registry, Kafka Rest Proxy, and Control Center.
• The valid Pod Overlay configuration is strategically merged with the pod spec inside the StatefulSet generated by CFK to form the final pod configuration for the application.

Configure Workloads Scheduling

You can control how the component pods are scheduled on Kubernetes nodes to get optimal performance out of Confluent components.

For example, you can configure pods not to be scheduled on the same node as other resource intensive applications, pods to be scheduled on dedicated nodes, or pods to be scheduled on the nodes with the most suitable hardware.

You can try it out here.

Index

Following index (not exhaustive) provides a list of scenarios available in this repository.

items	Tags
autogenerated-tls_only
blueprints	#control-plane/data-plane, #multiple-k8s-cluster
ccloud-connect-confluent-hub	#confluent-cloud-kafka
ccloud-integration
external-access-load-balancer-deploy	#loadbalancer
external-access-nodeport-deploy	#nodePort
external-access-static-host-based	#static-host
external-access-static-port-based	#static-port
hashicorp vault	#security #3rd-party
kafka-additional-listeners	#custom-listener
kraft_sasl_ssl_autogenerated
kubernetes-rbac	#k8s-rbac,#k8s-namespace
monitoring	#grafana
mtls-without-rbac
mtls-without-rbac	#mtls
multi-Region-Clusters (MRC)	#multiple-k8s-cluster
openshift-security
plaintext-basic-auth-Connect	#basic-auth
pod-overlay	#advanced-configuration
quickstart-deploy	#Beginner
replicator
schemalink
separate-listener-tls-rbac	#tls, #rbac
separate/mutiple kafka listener,	#tls
userprovided-tls_mtls-sasl_confluent-rbac
using-cert-manager	#tls, #3rd-party, #cert-manager

Troubleshooting

If you run into issues, don't forget to check the troubleshooting document.

confluent-kubernetes-examples's Issues

Provide Certificate Values as a Secret Reference

I am using the helm charts and I would like to be able to provide the kafka.tls.privkey, kafka.tls.fullchain and kafka.tls.cacerts as secret references (where the certs are provided by cert-manager). Right now this has to be a multi-step process to deploy (get certs, extract them into values.yaml, finally create confluent components). Allowing these to be provided via secrets would make the process simpler.

How to connect to kafka in openshift

I have enabled routing for kafka with a custom certificate, but I can't connect to kafka, can you help me

ksqldb is not accessible from control center in the static-host-based external access example

I was following the https://github.com/confluentinc/confluent-kubernetes-examples/blob/master/networking/external-access-static-host-based/confluent-platform-example.yaml to deploy confluent platform in AKS. However, after the cluster is deployed, the ksqldb is not accessible via the control center.
It seems the dependencies fields are missing from the controlcenter CR.
dependencies:
schemaRegistry:
url: http://schemaregistry.confluent.svc.cluster.local:8081
ksqldb:
- name: ksqldb
url: http://ksqldb.confluent.svc.cluster.local:8088
connect:
- name: connect
url: http://connect.confluent.svc.cluster.local:8083

Can help to confirm?

Nginx Rule file

High Priority, Blocker Issue.

While deploying the confluent platform on kubernetes version 1.22. I found that the nginx rule file provided in this repository is not supported by the latest stable version of kubernetes. So I have the new rule file which is perfectly tested and it is resolving the DNS, but I am not able to contribute to this repository.

missing confluent-platform-security-deployment.png

Looks like the hyper link to confluent-platform-security-deployment.png is broken on page:

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/security/secure-authn-encrypt-deploy

cp-server-connect version is not found when image is pulled

When I run kubectl apply kafka-connect.yaml
The connector remains forever in provisioning state
From the kubernetes logs:
It looks like it is not able to find the version that is referenced in the file (7.3.0):
Failed to pull image "confluentinc/cp-server-connect:7.3.0": rpc error: code = Unknown desc = context deadline exceeded
52m Warning Failed pod/connect-0 Error: ErrImagePull
7m12s Normal BackOff pod/connect-0 Back-off pulling image "confluentinc/cp-server-connect:7.3.0"

Change all FQDN / server names to a consistent variable over all the example

Values should be consistent over the examples repo.
For example: such as bootstrap , should be bootstrap:port and not FQDN

kafka rest API request failed: Unauthorized (0)

I followed all the steps provided for the example "production-secure-deploy" but restproxy is not starting up and i am getting Unauthorized error

4m15s Warning FailedToDiscover kafkarestclass/default waiting for at-least one kafka pod availability
5s Warning FailedToDiscover kafkarestclass/default kafka cluster [kafka] in not in running state
2m43s Warning Warning kafkarestclass/default https://kafka.confluent.svc.cluster.local:8090/kafka/v3 dial tcp 10.64.1.53:8090: connect: connection refused
100s Warning Warning kafkarestclass/default https://kafka.confluent.svc.cluster.local:8090/kafka/v3 dial tcp 10.64.2.60:8090: connect: connection refused
110s Warning Warning kafkarestclass/default https://kafka.confluent.svc.cluster.local:8090/kafka/v3 dial tcp 10.64.0.59:8090: connect: connection refused
15s Warning Warning kafkarestclass/default kafka rest API request failed: Unauthorized (0)

I did create the rest-credential from the file bearer.txt as defined in the document

kubectl create secret generic rest-credential --from-file=bearer.txt=$TUTORIAL_HOME/bearer.txt --from-file=basic.txt=$TUTORIAL_HOME/bearer.txt --namespace confluent

What could be the reason for this Unauthorized error

Schemaregistry - Unexpected 4xx health check response

I am seeing the following warning in my control center logs which I am unable to figure out.

It seems somewhere internally there is a healthcheck being done to a authorized endpoint. In this case a 401 should a good response.

[2022-01-03 16:33:03,175] WARN [armeria-common-worker-epoll-2-1] [creqId=a3f89d66, chanId=5ea82cd4, laddr=10.254.12.29:54770, raddr=schemaregistry.confluent.svc.cluster.local/10.254.12.22:8081][https://schemaregistry.confluent.svc.cluster.local:8081/#GET] Unexpected 4xx health check response: [:status=401, www-authenticate=basic realm="SchemaRegistry-Props", content-type=application/json;charset=iso-8859-1, content-length=43] A 4xx response generally indicates a misconfiguration of the client. Did you happen to forget to configure the HealthCheckedEndpointGroup's client options? (com.linecorp.armeria.client.endpoint.healthcheck.HttpHealthChecker)

It is only a warning, so not a huge issue.

In the hybrid/ccloud-JDBC-mysql tutorial, kafka.properties does not exist on connect pod

In order to complete the hybrid/ccloud-JDBC-mysql tutorial, I had to make a change at the very end.

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/hybrid/ccloud-JDBC-mysql#consume-from-the-topic-that-is-used-by-the-crd-created-connector

kafka.properties did not exist. To get it working, I changed it to consumer.properties.

Before:

kafka-avro-console-consumer \
--bootstrap-server CCLOUD:9092 \
--topic quickstart-jdbc-test \
--consumer.config /opt/confluentinc/etc/connect/kafka.properties \
--property schema.registry.url=SR_URL \
--property schema.registry.basic.auth.user.info=SR_USER:SR_SECRET \
--property basic.auth.credentials.source=USER_INFO \
--from-beginning

After:

kafka-avro-console-consumer \
--bootstrap-server CCLOUD:9092 \
--topic quickstart-jdbc-test \
--consumer.config /opt/confluentinc/etc/connect/consumer.properties \
--property schema.registry.url=SR_URL \
--property schema.registry.basic.auth.user.info=SR_USER:SR_SECRET \
--property basic.auth.credentials.source=USER_INFO \
--from-beginning

---

The following shows which files exist in that directory:

bash-4.4$ ls -l /opt/confluentinc/etc/connect/
total 16
-rw-r--r-- 1 1001 1001 3014 Aug 17 02:45 connect.properties
-rw-r--r-- 1 1001 1001  346 Aug 17 03:09 consumer.properties
-rw-r--r-- 1 1001 1001    0 Aug 17 02:45 disk-usage-agent.properties
-rw-r--r-- 1 1001 1001  761 Aug 17 02:45 jvm.config
-rw-r--r-- 1 1001 1001  441 Aug 17 02:45 log4j.properties

How to configure the Kafka cluster id as the same across all (point to the same Zookeeper)

Hi !

I'm trying to setup the multi-region-clusters as described in multi-region-clusters

In the process of Kafka set up, it is given as 'Configure the Kafka cluster id as the same across all (point to the same Zookeeper)'

How to achieve this?

Open ldap Error

nodeport does not update advertised listeners

We have updated to nodeport like so

kind: Kafka
spec:
...
  listeners:
    external:
      externalAccess:
        type: nodePort
        nodePort:
          host: 34.171.202.240
          nodePortOffset: 30000

But it looks like the advertised listeners are not updated as expected

$ kubectl -n confluent logs kafka-0 | grep advert
Defaulted container "kafka" out of: kafka, config-init-container (init)
        advertised.listeners = EXTERNAL://kafka-0.kafka.confluent.svc.cluster.local:9092,INTERNAL://kafka-0.kafka.confluent.svc.cluster.local:9071,REPLICATION://kafka-0.kafka.confluent.svc.cluster.local:9072
        advertised.listeners = EXTERNAL://kafka-0.kafka.confluent.svc.cluster.local:9092,INTERNAL://kafka-0.kafka.confluent.svc.cluster.local:9071,REPLICATION://kafka-0.kafka.confluent.svc.cluster.local:9072
        advertised.listeners = []

$ ./kcat.exe -b 34.171.202.240:30000 -t my-topic -C -v
%3|1668553116.364|FAIL|rdkafka#consumer-1| [thrd:kafka-0.kafka.confluent.svc.cluster.local:9092/0]: kafka-0.kafka.confluent.svc.cluster.local:9092/0: Failed to resolve 'kafka-0.kafka.confluent.svc.cluster.local:9092': No such host is known.  (after 19ms in state CONNECT)
% Fatal error at consume_cb:627:
% ERROR: Topic my-topic [0] error: Failed to query logical offset END: Local: Host resolution failure       

[FEATURE-REQUEST] Zookeeper-less Helmchart/ Configmap missing

Hey there,

due to the new release of confluent platform 6.2.1 and its including Apache Kafka 2.8 I'm curious and want to tryout the ZKless configuration options. All I found is the below but I'm stuck in the controller.quorum.voters part as its for every pod (3) node.id=-1 and kafka is crashing.

process.roles=broker,controller
controller.listener.names=CONTROLLER
listeners=PLAINTEXT://:9092,CONTROLLER://:19092
log.dirs=/tmp/server1/kraft-combined-logs
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
controller.quorum.voters=1@localhost:19092,2@localhost:19093,3@localhost:19094

Please add an article of config example to explain this part a bit further - Thank you very much :)

Cannot install using helm3 in vagrant

Good afternoon, I am trying to install CP using helm3 on a Vagrant virtual machine with a debian Buster image. When I run kubectl apply I get this error:

vagrant@buster:~/confluent-kubernetes-examples/quickstart-deploy$ kubectl apply -f confluent-platform-singlenode.yaml unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "Zookeeper" in version "platform.confluent.io/v1beta1" unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "Kafka" in version "platform.confluent.io/v1beta1" unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "Connect" in version "platform.confluent.io/v1beta1" unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "KsqlDB" in version "platform.confluent.io/v1beta1" unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "ControlCenter" in version "platform.confluent.io/v1beta1" unable to recognize "confluent-platform-singlenode.yaml": no matches for kind "SchemaRegistry" in version "platform.confluent.io/v1beta1"

vagrant@buster:~/confluent-kubernetes-examples/quickstart-deploy$ helm version
version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

What can i do?

Connect config file is not generated accordingly

cfk operator version: 2.50
OpenShift version: 4.9

I was trying to create a connect cluster in OpenShift that needs to connect to an on-prem Kafka Brokers. My CRD instance definition is as follows,

apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect-ocp
  namespace: kafka-poc
spec:
  license:
    globalLicense: true
  replicas: 1
  image:
    application: confluentinc/cp-server-connect:7.2.2
    init: confluentinc/confluent-init-container:2.5.0
  configOverrides:
    server:
      - group.id=ocp-connect-cluster-01      
  tls:
    autoGeneratedCerts: true
  authorization:
    type: rbac
  dependencies:
    kafka:
      bootstrapEndpoint: server1.domain:9093, server2.domain:9093
      authentication:
        type: plain
        jaasConfig:
          secretRef: credential
      tls:
        enabled: true
        ignoreTrustStoreConfig: true        
    mds:
      endpoint: https://server1.domain:8090, https://server2.domain:8090
      tokenKeyPair:
        secretRef: mds-token
      authentication:
        type: bearer
        bearer:
          secretRef: connect-mds-client
      tls:
        enabled: true
        ignoreTrustStoreConfig: true

It creates the connect-ocp-shared-config configmap as follows,

admin.bootstrap.servers=server1.domain:9073, server2.domain:9073
admin.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required metadataServerUrls="https://server1.domain:8090, https://server2.domain:8090" username="${file:/mnt/secrets/connect-mds-client/bearer.txt:username}" password="${file:/mnt/secrets/connect-mds-client/bearer.txt:password}";
admin.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
admin.sasl.mechanism=OAUTHBEARER
admin.security.protocol=SASL_SSL
bootstrap.servers=server1.domain:9073, server2.domain:9073
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.storage.replication.factor=3
config.storage.topic=kafka-poc.connect-ocp-configs
confluent.license=${file:/mnt/secrets/internal-confluent-operator-licensing/license.txt:license}
confluent.metadata.basic.auth.user.info=${file:/mnt/secrets/connect-mds-client/bearer.txt:username}:${file:/mnt/secrets/connect-mds-client/bearer.txt:password}
confluent.metadata.bootstrap.server.urls=https://server1.domain:8090, https://server2.domain:8090
confluent.metadata.http.auth.credentials.provider=BASIC
confluent.topic.replication.factor=3
connector.client.config.override.policy=All
consumer.bootstrap.servers=server1.domain:9073, server2.domain:9073
consumer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required metadataServerUrls="https://server1.domain:8090, https://server2.domain:8090" username="${file:/mnt/secrets/connect-mds-client/bearer.txt:username}" password="${file:/mnt/secrets/connect-mds-client/bearer.txt:password}";
consumer.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
consumer.sasl.mechanism=OAUTHBEARER
consumer.security.protocol=SASL_SSL
group.id=ocp-connect-cluster-01
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
listeners=https://0.0.0.0:8083
listeners.https.ssl.enabled.protocols=TLSv1.2
listeners.https.ssl.key.password=${file:/mnt/sslcerts/jksPassword.txt:jksPassword}
listeners.https.ssl.keystore.location=/mnt/sslcerts/keystore.jks
listeners.https.ssl.keystore.password=${file:/mnt/sslcerts/jksPassword.txt:jksPassword}
listeners.https.ssl.truststore.location=/mnt/sslcerts/truststore.jks
listeners.https.ssl.truststore.password=${file:/mnt/sslcerts/jksPassword.txt:jksPassword}
offset.flush.interval.ms=10000
offset.storage.replication.factor=3
offset.storage.topic=kafka-poc.connect-ocp-offsets
plugin.path=/usr/share/java,/usr/share/confluent-hub-components
producer.bootstrap.servers=server1.domain:9073, server2.domain:9073
producer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required metadataServerUrls="https://server1.domain:8090, https://server2.domain:8090" username="${file:/mnt/secrets/connect-mds-client/bearer.txt:username}" password="${file:/mnt/secrets/connect-mds-client/bearer.txt:password}";
producer.sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
producer.sasl.mechanism=OAUTHBEARER
producer.security.protocol=SASL_SSL
public.key.path=/mnt/secrets/mds-token/mdsPublicKey.pem
request.timeout.ms=20000
rest.advertised.listener=https
rest.extension.classes=io.confluent.connect.security.ConnectSecurityExtension
rest.servlet.initializor.classes=io.confluent.common.security.jetty.initializer.InstallBearerOrBasicSecurityHandler
retry.backoff.ms=500
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required metadataServerUrls="https://server1.domain:8090, https://server2.domain:8090" username="${file:/mnt/secrets/connect-mds-client/bearer.txt:username}" password="${file:/mnt/secrets/connect-mds-client/bearer.txt:password}";
sasl.login.callback.handler.class=io.confluent.kafka.clients.plugins.auth.token.TokenUserLoginCallbackHandler
sasl.mechanism=OAUTHBEARER
security.protocol=SASL_SSL
status.storage.replication.factor=3
status.storage.topic=kafka-poc.connect-ocp-status
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

Note that the bootstrap server port is 9073 instead of 9093. Also, the SASL mechanism and JAAS configs are related to OAUTHBEARER and not related to PLAIN.

What am I doing wrong?

secretRefs missing using confluent-platform-production-mtls.yaml

Following secretRefs were missing when using confluent-platform-production-mtls.yaml file:

11s         Warning   SecretRefIssue     connect/connect                      secretRef tls-connect not found
11s         Warning   SecretRefIssue     controlcenter/controlcenter     secretRef tls-controlcenter not found
11s         Warning   SecretRefIssue     kafka/kafka                               secretRef tls-kafka-external not found
11s         Warning   SecretRefIssue     ksqldb/ksqldb                           secretRef tls-ksqldb not found
11s          Warning   SecretRefIssue    kafka/kafka                                secretRef tls-kafka-internal not found
11s         Warning   SecretRefIssue     schemaregistry/schemaregistry   secretRef tls-schemaregistry not found
12s         Warning   SecretRefIssue     zookeeper/zookeeper             secretRef tls-zookeeper not found

I was able to install CP after creating the missing secrets for all the components. Can we please add the step to create the secrets for mtls authentication deployment?

Image cp-server cannot be replaced with cp-kafka

Hello,

I have tried to use CFK with cp-kafka instead of cp-server. According to https://hub.docker.com/r/confluentinc/cp-server the images should be replaceable (if no commercial features were used).

The configuration I have tried is:

apiVersion: platform.confluent.io/v1beta1
kind: Zookeeper
metadata:
  name: zookeeper
  namespace: confluent
spec:
  replicas: 1
  image:
    application: confluentinc/cp-zookeeper:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  dataVolumeCapacity: 5Gi
  logVolumeCapacity: 1Gi
  storageClass:
    name: managed-azuredisk-standard-lrs
  authentication:
    type: digest
    jaasConfig:
      secretRef: authentication-credentials
  tls:
    secretRef: kafka-tls
---
apiVersion: platform.confluent.io/v1beta1
kind: Kafka
metadata:
  name: kafka
  namespace: confluent
spec:
  replicas: 3
  image:
    application: confluentinc/cp-kafka:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  configOverrides:
    server:
      - authorizer.class.name=kafka.security.authorizer.AclAuthorizer
  dataVolumeCapacity: 5Gi
  storageClass:
    name: managed-azuredisk-standard-lrs
  metricReporter:
    enabled: false
  tls:
    secretRef: kafka-tls
  listeners:
    internal:
      authentication:
        type: plain
        jaasConfig:
          secretRef: authentication-credentials
      tls:
        enabled: true
    external:
      authentication:
        type: plain
        jaasConfig:
          secretRef: authentication-credentials
      tls:
        enabled: true
      externalAccess:
        type: loadBalancer
        loadBalancer:
          domain: my.domain
          advertisedPort: 30129
          brokerPrefix: kafka-
          bootstrapPrefix: kafka
  authorization:
    type: simple
    superUsers:
      - User:admin
  dependencies:
    zookeeper:
      endpoint: zookeeper.confluent.svc.cluster.local:2182
      authentication:
        type: digest
        jaasConfig:
          secretRef: authentication-credentials
      tls:
        enabled: true

With authentication-credentials secret containing (with the according content):

• digest-users.json
• digest.txt
• plain-users.json
• plain.txt

With cp-server image this works. But with cp-kafka, the follwing exception is thrown:

[ERROR] 2023-01-19 10:58:23,889 [main] kafka.server.KafkaServer error - [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown
org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for io.confluent.kafka.server.plugins.auth.FileBasedLoginModule
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184)
	at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192)
	at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)
	at kafka.server.BrokerToControllerChannelManagerImpl.newRequestThread(BrokerToControllerChannelManager.scala:188)
	at kafka.server.BrokerToControllerChannelManagerImpl.<init>(BrokerToControllerChannelManager.scala:167)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:286)
	at kafka.Kafka$.main(Kafka.scala:109)
	at kafka.Kafka.main(Kafka.scala)
Caused by: javax.security.auth.login.LoginException: No LoginModule found for io.confluent.kafka.server.plugins.auth.FileBasedLoginModule
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:731)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:581)
	at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60)
	at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:62)
	at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:105)
	at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:170)
	... 7 more

As it turns out, by letting the CFK operator manage the users, the FileBasedLoginModule seems to be required.

Is there a workaround for this? Has someone tried to use cp-kafka in combination with CFK?

Cp-helm-charts for production

Hi Team,

Can we use cp-helm-charts for production use, If not what are the changes to make it for production use from below link.

https://github.com/confluentinc/cp-helm-charts

I am planning to use below services from confluent community images.

kafka-connect
kafka
ksql-server
schema-registry
zookeeper

Thanks,
Bharath.

In the quickstart the integration between KSQL, SR en the brokers is not working

In the quickstar, in order to better showcase the integration of the components:

For the brokers
The kafka-brokers needs the following configuration under spec

  configOverrides:
    server:
    - confluent.schema.registry.url=http://schemaregistry.confluent.svc.cluster.local:8081  

This will allow you to be able to set the confluent_value_validation on a topic

For KsqlDb
KsqlDb needs to integrate with the broker and connect with schema registry:

  dependencies:
    kafka:
      bootstrapEndpoint: kafka.confluent.svc.cluster.local:9071
  configOverrides:
    server:
    - ksql.schema.registry.url=http://schemaregistry.confluent.svc.cluster.local:8081  

Got error related to secrets when applied confluent-platform-production.yaml

I have applied what i got from the repo but unable to pass those errors can I know to fix these issues?

Getting permission denied message on RKE 1.1.19

I have deployed confluent-operator as per this document
https://docs.confluent.io/operator/current/co-quickstart.html

--version, -v print the version
[root@tfancbc-ibmdevel-1 kafka-pv]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
10.101.216.91 Ready controlplane,etcd 20d v1.18.20
10.101.216.92 Ready worker 20d v1.18.20
10.101.216.93 Ready worker 20d v1.18.20
[root@tfancbc-ibmdevel-1 kafka-pv]# kubectl get pod
NAME READY STATUS RESTARTS AGE
confluent-operator-78674949c9-hc4v2 1/1 Running 0 30m
connect-0 0/1 CrashLoopBackOff 8 29m
zookeeper-0 0/1 CrashLoopBackOff 10 29m
zookeeper-1 0/1 CrashLoopBackOff 10 29m
zookeeper-2 0/1 CrashLoopBackOff 6 10m
[root@tfancbc-ibmdevel-1 kafka-pv]#

[root@tfancbc-ibmdevel-1 kafka-pv]# rke --version
rke version v1.1.19
[root@tfancbc-ibmdevel-1 kafka-pv]#

But while bringing up zookeeper I am getting below issue

[root@tfancbc-ibmdevel-1 kafka-pv]# kubectl logs zookeeper-2

• /mnt/config/zookeeper/bin/run
  ===> User
  uid=1001 gid=0(root) groups=0(root),1001
  ===> Load zookeeper operator scripts from path /mnt/config/zookeeper/bin
  /mnt/config/zookeeper/bin/configure: line 13: /mnt/data/data/myid: Permission denied
  [root@tfancbc-ibmdevel-1 kafka-pv]#

This is security context set in pod(default value)
securityContext:
fsGroup: 1001
runAsNonRoot: true
runAsUser: 1001

This the sc and pv details
[root@tfancbc-ibmdevel-1 kafka-pv]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
slow (default) kubernetes.io/gce-pd Delete Immediate false 11d
[root@tfancbc-ibmdevel-1 kafka-pv]# kubectl get pv | grep confluent
data-zookeeper-1 10Gi RWO Retain Bound confluent/txnlog-zookeeper-0 slow 29m
data-zookeeper-2 10Gi RWO Retain Bound confluent/data-zookeeper-1 slow 29m
data-zookeeper-3 10Gi RWO Retain Bound confluent/data-zookeeper-2 slow 29m
log-zookeeper-1 10Gi RWO Retain Bound confluent/txnlog-zookeeper-2 slow 29m
log-zookeeper-2 10Gi RWO Retain Bound confluent/data-zookeeper-0 slow 29m
log-zookeeper-3 10Gi RWO Retain Bound confluent/txnlog-zookeeper-1 slow 29m

controlcenter Pod didn't got deployed using production secure deploy steps

https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/security/production-secure-deploy

I have followed above step and didn't miss any steps but I don't see a pod or deployment before doing a kubectl portforward for control center.

Can I know if there anything apart from those steps need to be deployed in the cluster?

All i see is the below pods before i run the port forward command.

Control Center doesn't work when ui.basepath is set

As mentioned in
confluentinc/control-center-images#53 (comment)

If we set confluent.controlcenter.ui.basepath=/abc/ control center doesn't start. Assumption liveness/readiness probe does not respect specified base path.

Control Center with basic auth kafka connect dependency

This configuration doesn't work:

---
apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect
  namespace: confluent
spec:
  replicas: 1
  authentication:
    type: basic
    basic:
      secretRef: basicsecret
  tls:
    autoGeneratedCerts: true
  image:
    application: confluentinc/cp-server-connect:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  dependencies:
    kafka:
      bootstrapEndpoint: kafka:9071

---
apiVersion: platform.confluent.io/v1beta1
kind: ControlCenter
metadata:
  name: controlcenter
  namespace: confluent
spec:
  replicas: 1
  image:
    application: confluentinc/cp-enterprise-control-center:7.3.0
    init: confluentinc/confluent-init-container:2.5.0
  dataVolumeCapacity: 10Gi
  dependencies:
   connect:
      url: https://connect.confluent.svc.cluster.local:8083
      authentication:
        type: basic
        basic:
          secretRef: basicsecret

I am seeing the following WARN logs:

[2023-02-07 21:05:32,689] WARN [armeria-common-worker-epoll-2-2] [creqId=xxxxxxxx, chanId=xxxxxxxx, laddr=10.133.151.141:53104, │
│  raddr=connect/x.x.x.x:8083][https://connect:8083/v1/metadata/id#GET] Unexpected 4xx health check respo │
│ nse: [:status=401, date=Tue, 07 Feb 2023 21:05:32 GMT, content-type=application/json, content-length=32] A 4xx response generally indicates a mis │
│ configuration of the client. Did you happen to forget to configure the HealthCheckedEndpointGroup's client options? (com.linecorp.armeria.client. │
│ endpoint.healthcheck.HttpHealthChecker)

I thought I would find the answer in the examples; however, the plaintext-basic-auth-Connect example configures connect with basic auth but conveniently does not add specify connect as a dependency. Is this an unsupported configuration?

I know the Kafka Connect basic auth rest endpoint is valid and know Control Center can talk to Connect over unauthenticated/TLS.

# using an self-signed autogenerated cert
$ curl --insecure -u connect:xxxxxxxxxxxxxxxx https://connect:8083/v1/metadata/id#GET | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0   3424      0 --:--:-- --:--:-- --:--:--  3424
{
  "id": "",
  "scope": {
    "path": [],
    "clusters": {
      "kafka-cluster": "XXXXXXXXXXXXXXXX",
      "connect-cluster": "connect"
    }
  }
}

Is confluent-rolebinding supported for Confluent Cloud?

Resources such as Topics and schemas can be provisioned using the CustomResources: KafkaTopics and Schemas. However i am failing to provision Rolebindings for Confluent Cloud:

apiVersion: platform.confluent.io/v1beta1
kind: ConfluentRolebinding
metadata:
  name: kafka-rb
spec:
  principal:
    type: user
    name: u-test
  clustersScopeByIds:
    kafkaClusterId: lkc-0test
  kafkaRestClassRef:
    name: default
    namespace: confluent
  resourcePatterns:
  - name: "*"
    patternType: PREFIXED
    resourceType: Topic
  role: DeveloperRead

apiVersion: platform.confluent.io/v1beta1
kind: KafkaRestClass
metadata:
  name: default
  namespace: confluent
spec:
  kafkaRest: 
    endpoint: https://api.confluent.cloud/iam/v2
    kafkaClusterID: lkc-0test
    authentication:
      bearer:
        secretRef: rest-creds
      type: bearer

In the controller logs:

"controllers/confluentrolebinding_controller.go:130","msg":"apply failed","name":"kafka","namespace":"confluent","error":"kafka rest API request failed: Not Found (0)"

Has anyone actually tried to create a rolebinding that gets provisioned in confluent cloud? What am i missing in the configuration?

Exception while loading Zookeeper JAAS

While following the step mentioned in production-secure-deploy to enable authentication to zookeeper via digest i am getting below error.

creds-zookeeper-sasl-digest-users.json

{
"kafka": "kafka-secret"
}

creds-kafka-zookeeper-credentials.txt

username="kafka"
password="kafka-secret"

[ERROR] 2021-08-30 07:48:56,286 [main] kafka.server.KafkaServer error - Fatal error during KafkaServer startup. Prepare to shutdown
org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/mnt/secrets/digest-jaas.conf, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]

Unable to canonicalize address

Hey there

I have a little problem here with the single-node example. There is an error in Kafka pod :

[WARN] 2022-04-26 14:29:47,008 [main-SendThread(zookeeper.confluent.svc.cluster.local:2181)] org.apache.zookeeper.ClientCnxn run - Session 0x0 for sever zookeeper.confluent.svc.cluster.local:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.IllegalArgumentException: Unable to canonicalize address zookeeper.confluent.svc.cluster.local:2181 because it's not resolvable
        at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
        at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1161)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1210)
[INFO] 2022-04-26 14:29:47,273 [main] kafka.zookeeper.ZooKeeperClient info - [ZooKeeperClient Kafka server] Closing.
[ERROR] 2022-04-26 14:29:48,112 [main-SendThread(zookeeper.confluent.svc.cluster.local:2181)] org.apache.zookeeper.client.StaticHostProvider resolve - Unable to resolve address: zookeeper.confluent.svc.cluster.local:2181
java.net.UnknownHostException: zookeeper.confluent.svc.cluster.local

Error messages :

Unable to canonicalize address zookeeper.confluent.svc.cluster.local:2181 because it's not resolvable
Unable to resolve address: zookeeper.confluent.svc.cluster.local:2181

I checked my zookeper ... it's good and works without a problem.

Also, check DNS using dnsutils :

$ kubectl -n default exec -it dnsutils -- nslookup zookeeper.confluent.svc.cluster.local

Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   zookeeper.confluent.svc.cluster.local
Address: 192.168.0.111

What can I do?

Add example on Pod's Workload Scheduling

Why

By default, multiple pods can land in the same node (or same zone). Providing a quick example to set node awareness helps adopting CFK correctly.

kafka not able to connect to zookeeper

When this command executed:
kubectl apply -f https://raw.githubusercontent.com/confluentinc/confluent-kubernetes-examples/master/quickstart-deploy/confluent-platform.yaml

Confluent operator log:
{"level":"ERROR","time":"2023-04-28T21:01:11.301Z","name":"kafka","caller":"log/log.go:35","msg":"apply failed","name":"kafka","namespace":"mnk","error":"no zk endpoint or service discovery configuration found for dependency"}
Logs from 4/28/2023, 1:57:35 AM

So kafka installation do not start and other components as well waiting for kafka.

Ability to create a monitoring solution quickly

The current examples require you to understand Prometheus, Grafana, and JMX in some depth to make it work. We would like an example where a setup can be achieved within several minutes.

Can't connect with Flink client from operator via minikube with loadbalancer

I am evaluating the confluent operator as a developer. I have been trying to connect to the kafka server via my local IDE using Apache Flink, for a few days. I have gone through some of the networking examples and configured load balancers similar to what is shown in https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/external-access-load-balancer-deploy .

I am deploying confluent platform on minikube (mentioned in the examples). I noticed in my debugging logs that it looks like the kafka client is trying to connect to b0.minikube.domain. One thing I find interesting regarding this is the b0 host alias. It also looks like it's trying to use port 9092 when the loadbalncer port via minikube is configured otherwise. I don't see this hostname b0 configured explicitly anywhere in confluent kubernetes helm charts.

21:30:01,196 DEBUG org.apache.kafka.clients.NetworkClient                       [] - [Producer clientId=producer-9] Initialize connection to node b0.minikube.domain:9092 (id: 0 rack: 0) for sending metadata request
21:30:01,196 WARN  org.apache.kafka.clients.NetworkClient                       [] - [Producer clientId=producer-9] Error connecting to node b0.minikube.domain:9092 (id: 0 rack: 0)
java.net.UnknownHostException: b0.minikube.domain
    at [java.net](http://java.net/).InetAddress$CachedAddresses.get(InetAddress.java:797) ~[?:?]
    at [java.net](http://java.net/).InetAddress.getAllByName0(InetAddress.java:1519) ~[?:?]
    at [java.net](http://java.net/).InetAddress.getAllByName(InetAddress.java:1378) ~[?:?]
    at [java.net](http://java.net/).InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
    at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1161) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1049) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:558) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) ~[kafka-clients-3.2.3.jar:?]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) ~[kafka-clients-3.2.3.jar:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]

add a note for autogeneratedcerts scenario

Encountered secret missing error when using confluent-platform-production-autogeneratedcerts.yaml file to deploy CP:

Output from kubectl get events

23s Warning KeyInSecretRefIssue zookeeper/zookeeper secret ca-pair-sslcerts missing in namespace confluent. Please create a secret object accordingly

Created a ca-pair-sslcerts secret following the link resolved the issue.

It'd be helpful to add a note for autogeneratedcerts scenario in the security/production-secure-deploy/README.rst

How to set external listeners like # EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))?

In https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/networking/kafka-additional-listeners it shows how to set custom listeners. But previously we set external listeners like https://www.confluent.io/blog/kafka-listeners-explained/ .

Specifically setting https://github.com/confluentinc/cp-helm-charts/blob/524608ded7c8aecf2540933c267e990e3c917f04/charts/cp-kafka/values.yaml#L58 allowed us to create an external listener for each broker deployed from the helm chart each with a unique port. This worked well for my local environment. Then how do we accomplish this with the new syntax?

Principal Mapping Rules in the Kafka ACL Examples Remove the Leading "S" from Principal Names

When running some of the ACL authorizer examples (for example, https://github.com/confluentinc/confluent-kubernetes-examples/blob/master/security/userprovided-tls_mtls-sasl_confluent-rbac/confluent-platform-mtls-sasl-rbac.yaml), the principal mapping rule needs to be updated to escape the string to prevent issues with principals that have a CN starting with "s."

Errors were observed specific to the "sr" user, where the Kafka brokers were showing errors in the logs that "User: r" did not have the proper authorizations (the s was getting stripped). Updating the rule to the syntax below resolved the issue

        principalMappingRules:
          - RULE:.*CN[\\s]?=[\\s]?([a-zA-Z0-9.]*)?.*/$1/

configmap changes been ignored and resetted

helm version: helm.sh/chart: confluent-for-kubernetes-0.174.13
docker-image version: confluentinc/cp-server-operator:6.1.2.1
kube-version: v1.21.3

Setup:

k8s with deployed confluent-operator via helm and running kafka, zk, connect + controlcenter while using the quickstart-deploy example manifest.

I wanted to add the ability to kafka and controlcenter to self-balance as described here > https://docs.confluent.io/platform/6.1.2/kafka/sbc/index.html#enabling-self-balancing-clusters

When I try to add the option "# Kafka REST endpoint URL":
confluent.controlcenter.streams.cprest.url=http://kafka.confluent.svc.cluster.local:8090

.. the configmap gets saved with:

root@controlplane1:~/confluent-kubernetes-examples# kubectl -n confluent edit configmaps controlcenter-shared-config
configmap/controlcenter-shared-config edited

https://docs.confluent.io/platform/current/kafka/sbc/configuration_options.html#required-configurations-for-c3-short

But nothing changes and when I reopen the configmap my changes are gone and been actually ignored.

Expected:

I can edit the configmap. save it and controlcenter restarts with the edited configmap.

Update kubectl cmds to include "-n confluent"

Can we update all the commands to be consistent and include "-n confluent" for all kubectl commands.

Operator no longer able to access Kafka REST API when it is secured

Reproducing the issue:

Deploy Confluent Operator
Deploy Secure Deployment https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/security/production-secure-deploy
Create a Topic Using CRD

Error messages registered in Operator Pod

{"level":"error","ts":1622449532.9655285,"logger":"kafkatopic","caller":"controller/controller.go:244","msg":"apply failed","name":"elastic-1","namespace":"confluent","error":"kafka rest API request failed: Unauthorized (0)"}

namespace step missing for production-recommended-secure-setup

Can we please add a step to create a namespace and setting the namespace for the k8s context on production-recommended-secure-setup page?

• kubectl config set-context --current --namespace confluent
• https://docs.confluent.io/operator/current/co-quickstart.html#step-1-create-a-namespace

Another suggestion would be to add the --namespace <namespace> to the commands used on secure setup page.

Configure locationType: url correctly for kubernetes

Hello!
I am trying to configure kafka connect for kubernetes.
In docs I've found how to add locationType: confluenthub and url.
Confluenthub config works great but I am trying also to use url from maven repository. How can I do that correctly ?

apiVersion: platform.confluent.io/v1beta1
kind: Connect
metadata:
  name: connect
  namespace: confluent
spec:
  replicas: 1
  image:
    application: confluentinc/cp-server-connect:7.0.1
    init: confluentinc/confluent-init-container:2.2.0-1
  configOverrides:
    server:
      - config.storage.replication.factor=1
      - offset.storage.replication.factor=1
      - status.storage.replication.factor=1
 build:
    type: onDemand
    onDemand:
      plugins:
        locationType: confluentHub #works great
        confluentHub:
          - name: kafka-connect-jdbc
            owner: confluentinc
            version: 10.2.6
          - name: kafka-connect-oracle-cdc
            owner: confluentinc
            version: 1.5.0
          - name: debezium-connector-mysql
            owner: debezium
            version: 1.7.1
          - name: debezium-connector-postgresql
            owner: debezium
            version: 1.7.1
      storageLimit: 4Gi
  podTemplate:
    resources:
      requests:
        cpu: 200m
        memory: 1024Mi
    probe:
      liveness:
        periodSeconds: 180 #DONT CHANGE THIS
        failureThreshold: 5
        timeoutSeconds: 500
    podSecurityContext:
      fsGroup: 1000
      runAsUser: 1000
      runAsNonRoot: true

What I did try.

locationType: url #NOT WORKING. NO IDEA HOW TO CONFIGURE THAT
    url:
       - name: mysql-connector-java
         archivePath: https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.26/mysql-connector-java-8.0.26.jar
         checksum: sha512sum    #definitely wrong

Also I tried making docker image but it doesn't work. (Pod is Unready all the time).

[Request] add userSearchBase and groupSearchBase to security/RBAC config

As an additional hint for MDS / Confluent Server Authorizer's LDAP configuration, add the following parameters with their defaults (to keep same functionality within current examples):

• userSearchScope: 1
• groupSearchScope: 1

These configuration should go into services.mds.provider.ldap.configurations section (here).

The user may continue to refer to the LDAP Authorizer documentation for clarification on these properties.

Use http/s proxy to download connect plugins into the init container from the internet

Currently, I don't find any method to inject forward proxy config into the init container so that it can download connect plugins from the internet. The podTemplate.envVars can only inject environment variables into the main container.

remove "k" alias from kubectl cmd

In the section to update authenticated users theres an alias "k" in the final line:

--save-config --dry-run=client -oyaml | k apply -f -

We should replace this with kubectl
Ref: https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/security/production-secure-deploy#appendix-update-authentication-users

Set appProtocol for service resources (compatibility with service mesh)

Hi. Currently there is no way to specify the appProtocol for services created by the operator. Unfortunately Microsoft's service mesh (Open Service Mesh) assumes the appProtocol is HTTP, which breaks Kafka and ZooKeeper. Please add appProtocol to all service ports or offer a way to customize it (e.g. something like podTemplate in the CRD but for the services).

Application protocol

FEATURE STATE: Kubernetes v1.20 [stable]

The appProtocol field provides a way to specify an application protocol for each Service port. The value of this field is mirrored by the corresponding Endpoints and EndpointSlice objects.

This field follows standard Kubernetes label syntax. Values should either be IANA standard service names or domain prefixed names such as mycompany.com/my-custom-protocol.

• Kubernetes docs: https://kubernetes.io/docs/concepts/services-networking/service/#application-protocol
• Open Service Mesh docs: https://release-v1-2.docs.openservicemesh.io/docs/guides/app_onboarding/app_protocol_selection/

e.g.

apiVersion: v1
kind: Service
metadata:
  name: kafka
spec:
  ports:
  - name: external
    port: 9092
    protocol: tcp
    appProtocol: tcp
    targetPort: 9092
  - name: http-external
    port: 8090
    protocol: tcp
    appProtocol: http
    targetPort: 8090
  - name: internal
    port: 9071
    protocol: tcp
    appProtocol: tcp
    targetPort: 9071
  - name: jmx
    port: 7203
    protocol: tcp
    appProtocol: tcp
    targetPort: 7203
  - name: jolokia
    port: 7777
    protocol: tcp
    appProtocol: http
    targetPort: 7777
  - name: prometheus
    port: 7778
    protocol: tcp
    appProtocol: http
    targetPort: 7778
  - name: replication
    port: 9072
    protocol: tcp
    appProtocol: tcp
    targetPort: 9072

apiVersion: v1
kind: Service
metadata:
  name: zookeeper
spec:
  ports:
  - name: client
    port: 2181
    protocol: tcp
    appProtocol: tcp
    targetPort: 2181
  - name: jmx
    port: 7203
    protocol: tcp
    appProtocol: tcp
    targetPort: 7203
  - name: jolokia
    port: 7777
    protocol: tcp
    appProtocol: http
    targetPort: 7777
  - name: leader-election
    port: 3888
    protocol: tcp
    appProtocol: tcp
    targetPort: 3888
  - name: peer
    port: 2888
    protocol: tcp
    appProtocol: tcp
    targetPort: 2888
  - name: prometheus
    port: 7778
    protocol: tcp
    appProtocol: http
    targetPort: 7778

Istio ingress gateway to access Kafka examples?

Hello, I'm trying to set up confluent platform and make kafka accessible via istio's ingress gateway, but I can't get it to work and I cannot find anywhere in documentation or submitted issues of anyone attempting to make it work behind istio ingress gateway.

I get these messages from fluentbit with the kafka conf set to send to brokers via the value:

    kafka:
      enable: true
      brokers: "kafka.mydomain.com:80"
      topics: ks-log

for reference it is taken from here

https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/values.yaml#L124

here are the messages I get

[2022/06/17 19:29:01] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2022/06/17 19:29:01] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2022/06/17 19:29:01] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2022/06/17 19:29:01] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2022/06/17 19:29:01] [ warn] [output:kafka:kafka.0] message delivery failed: Local: Message timed out
[2022/06/17 19:29:01] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:app]: fluent-bit#producer-1: kafka.mydomain.com:80/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 9999ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)
[2022/06/17 19:29:04] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: 1/1 brokers are down
[2022/06/17 19:29:11] [error] [filter:kubernetes:kubernetes.1] kubelet upstream connection error
[2022/06/17 19:29:15] [ info] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: kafka.9tlat6vp.wooden-proton.com:80/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 10001ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)
[2022/06/17 19:29:15] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: 1/1 brokers are down
[2022/06/17 19:29:21] [error] [filter:kubernetes:kubernetes.1] kubelet upstream connection error
[2022/06/17 19:29:26] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: 1/1 brokers are down
[2022/06/17 19:29:31] [error] [filter:kubernetes:kubernetes.1] kubelet upstream connection error
[2022/06/17 19:29:37] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: 1/1 brokers are down
[2022/06/17 19:29:41] [error] [filter:kubernetes:kubernetes.1] kubelet upstream connection error
[2022/06/17 19:29:48] [ info] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: kafka.9tlat6vp.wooden-proton.com:80/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 10000ms in state APIVERSION_QUERY, 3 identical error(s) suppressed)
[2022/06/17 19:29:48] [error] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:kafka.mydomain.com:80/bootstrap]: 1/1 brokers are down
[2022/06/17 19:29:51] [error] [filter:kubernetes:kubernetes.1] kubelet upstream connection error

here are my istio objects:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: confluent-gateway
  namespace: confluent
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - kafka.mydomain.com
    - control.mydomain.com
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: control-v-service
  namespace: confluent
spec:
  hosts:
  - control.mydomain.com
  gateways:
  - confluent-gateway
  http:
  - route:
    - destination:
        host: controlcenter
        port:
          number: 9021
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: kafka-v-service
  namespace: confluent
spec:
  hosts:
  - kafka.mydomain.com
  gateways:
  - confluent-gateway
  http:
  - route:
    - destination:
        host: kafka
        port:
          number: 9092

It is noteworthy that this works to get me into the control center following this model, but kafka brokers dont seem to be happy with the attempt. I also tried to change 9092 to 9097 (i believe in the quickstart it uses the internal port) but both have the same result.

Since confluent for kubernetes is obviously trying to work with Kubernetes I think it might be good to have an example in these kubernetes examples for istio ingress to work since Istio is so popular.

I'm not sure what else to try here. I thought maybe try looking at security.protocol configuration, but in the quickstart the client it set to plaintext, and according to librdkafka, which is what fluentbit uses

https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

plaintext is the default so I dont think its because its trying to SSL or anything like that.

Dashboards don't match metric attributes

It looks like the dashboards in this repo do not match the exported metrics.

For instance - the grafana query:

histogram_quantile(0.5, sum(rate(controller_runtime_reconcile_time_seconds_bucket{app="confluent-operator",
confluent_platform=~"true",kubernetes_namespace=~"$namespace", kubernetes_pod_name=~"$pod_name"}[5m])) by (le, controller))

uses metric controller_runtime_reconcile_time_seconds_bucket

while metric attributes differ from query:

controller_runtime_reconcile_time_seconds_bucket{controller="confluentrolebinding", endpoint="http-metric", instance="172.16.107.115:7778", job="confluent-operator", le="+Inf", namespace="confluent", pod="confluent-operator-55695f4b58-c42r7", service="confluent-operator"}

Is it possible to get these dashboards to work with current set of exported metrics?

Unused file in userprovided-tls_mtls_kafka-acls

It appears the file kafka-client.txt within userprovided-tls_mtls_kafka-acls never gets used and can be removed. This is a very effective example btw, worked great!

Also the directory has a mix of hyphens and underscores, might want to standardize.

Another idea that could be worth mentioning is simply adding all the component principals to the super.users list

quickstart-deploy's producer not starting

Expected

When kubectl apply -f producer-app-data-singlenode.yaml is executed, it should start the producer application.

*Observed

When the command is ran, the below error occurs and does not start the producer application.

Error log

> kubectl describe pod elastic-0

...

Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m15s                  default-scheduler  Successfully assigned confluent/elastic-0 to minikube
  Normal   Pulled     5m13s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.543359417s
  Normal   Pulled     5m11s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.564164709s
  Normal   Pulled     4m57s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.535623417s
  Normal   Pulled     4m44s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.582455626s
  Normal   Pulled     4m30s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.599102876s
  Normal   Pulled     4m17s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.563162001s
  Normal   Pulled     4m1s                   kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.548017126s
  Warning  Failed     3m49s (x8 over 5m13s)  kubelet            Error: container has runAsNonRoot and image has non-numeric user (appuser), cannot verify user is non-root (pod: "elastic-0_confluent(9577d9cd-d7c8-426a-b78a-93613d1763ed)", container: elastic)
  Normal   Pulled     3m49s                  kubelet            Successfully pulled image "confluentinc/cp-kafka:latest" in 1.552463959s
  Normal   Pulling    3m34s (x9 over 5m15s)  kubelet            Pulling image "confluentinc/cp-kafka:latest"

Additional info

• This was ran using minikube, specifically using this command to start minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy --addons=pod-security-policy --kubernetes-version v1.24.0

Environment

Minikube: v1.28.0
Kubernetes: v1.24.0

There is no way to not use nodePortOffset?

Hi,
I want to deploy confluent platform and kafka has external access with nodeport type. nodePortOffset is required and all N+1 nodeports shouldn't be already allocated by another app. Is there any way to automatically choose the ports not already allocated and they don't be sequential.

Typo in the container image name

confluent-kubernetes-examples/hybrid/clusterlink/mtls_source_cluster/zk-kafka-destination.yaml

Line 28 in af3ea4d

init: confluentinc/confluent-init-containerr:2.2.0

There is an extra "r" on the init container name for kafka deployment. This will cause an error when deploying on destination