confluentinc / ksql-images Goto Github PK
View Code? Open in Web Editor NEWThis project forked from mikebin/ksql-images
KSQL platform docker images
License: Apache License 2.0
This project forked from mikebin/ksql-images
KSQL platform docker images
License: Apache License 2.0
When i am using confluentinc/ksqldb-server:0.23+
and it support timestamp data type.
But cp-ksqldb-server:6.1.0
is not supporting and i got
error resolve timestamp
error
First off, thank you for these wonderful container images.
When running the ksqlDB quickstart example locally on my MacBook Pro M1 (which has an arm64 processor), I experienced problems with performance, resulting in connection problems when trying to connect from the ksql-cli to the ksql-server. Also, queries/commands on the cli (when it does connect to the ksqlDB server) sometimes just freeze and if they execute, they take longer than necessary.
For clarification, in Docker Desktop it shows the orange exclamation mark with the ksqldb-cli image based containers:
I have identified that this is caused by the fact that for the ksqldb-server image no linux/arm64
version is available (in contrast to other images like images for kafka and zookeeper). Docker Desktop tries to 'help' by using QEMU under the hood to allow this image to 'run', which it does, but with a terrible performance.
Could you please add support for the ARM64 architecture for this ksqlDB server image? That would allow me and other Apple M1 users to use the quickstart in a local fashion as intended.
Hi, I wanted to try the KSQL migration tool but looks like the confluentinc/cp-ksqldb-server
image does not embed the ksqldb-tools jar...
By comparing the following pom's
ksqlsb-tools
dependency is not present in the first but it is in the latter.I can only find version 6.0.0 on docker hub even though I think version 6.1.0 should be in docker hub.
https://hub.docker.com/r/confluentinc/cp-ksqldb-server/tags?page=1&ordering=last_updated&name=6
Am I doing something wrong?
PR #22, shortly prior to the CP5.4 release, added a new self-health-check function to the ksql-server image. This new function is invoked every 5 seconds by default (see https://github.com/confluentinc/ksql-images/blob/master/cp-ksql-server/Dockerfile.deb9#L41-L42) and works by posting a show topics;
query to the running ksql server.
There are several problems with this:
1 - it is inconsistent with all the other CP docker images, with the possible exception of Connect which looks to have something similar added. This means that when you bring up a CP stack with docker-compose up
and then check it's status with docker-compose up
, only ksql-server (and ppossibly Connect) report their status as Up (healthy)
. This gives the misleading impression that the other platform components are NOT healthy - a check like this needs to be all or none of the images.
2 - running show topics;
on a "real" cluster is potentially an expensive and time-consuming operation - certainly more than should be running every 5 seconds from every server.
3 - the "woohoo ksql is up!" message is less than professional looking
4 - the check runs too frequently (every 5 seconds by default) and spams the docker logs with verbose output at default logging levels
5 - this check is undocumented - you have to spelunk this private repo code to figure out how to adjust or disable it.
Example log output:
ksql-server | [2020-01-17 01:46:29,759] INFO 127.0.0.1 - - [17/Jan/2020:01:46:29 +0000] "POST /ksql HTTP/1.1" 200 4000 81 (io.confluent.rest-utils.requests:62)
ksql-server | [2020-01-17 01:46:34,982] INFO Received: KsqlRequest{ksql='SHOW TOPICS;', streamsProperties={}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:200)
ksql-server | [2020-01-17 01:46:34,984] INFO AvroDataConfig values:
ksql-server | connect.meta.data = true
ksql-server | enhanced.avro.schema.support = false
ksql-server | schemas.cache.config = 1000
ksql-server | (io.confluent.connect.avro.AvroDataConfig:347)
18:19:52 + cd ksql-images
18:19:52 + git checkout 5.4.4-post
18:19:52 Already on '5.4.4-post'
18:19:52 + mvn install:install-file -Dpackaging=pom -Dfile=pom.xml -DpomFile=pom.xml -Dmaven.test.skip=true
18:19:52 [INFO] Scanning for projects...
18:19:54 [WARNING]
18:19:54 [WARNING] Some problems were encountered while building the effective model for io.confluent.ksql:cp-ksql-cli๐ซ5.4.4
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: ${project.groupId}:ksql-functional-tests:jar -> duplicate declaration of version ${project.version} @ line 101, column 14
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: ${project.groupId}:ksql-functional-tests:jar -> duplicate declaration of version ${project.version} @ line 107, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.kafka:kafka-clients:jar:test -> duplicate declaration of version ${kafka.version} @ line 113, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.kafka:kafka-streams-test-utils:jar -> duplicate declaration of version ${kafka.version} @ line 121, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-metastore:test-jar -> duplicate declaration of version ${project.version} @ line 127, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-engine:test-jar -> duplicate declaration of version ${project.version} @ line 134, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-common:test-jar -> duplicate declaration of version ${project.version} @ line 141, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.hamcrest:hamcrest-all:jar -> duplicate declaration of version 1.3 @ line 148, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: junit:junit:jar -> duplicate declaration of version 4.12 @ line 155, column 21
18:19:54 [WARNING]
18:19:54 [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
18:19:54 [WARNING]
18:19:54 [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
18:19:54 [WARNING]
The following image(5.4.11-1) is vulnerable to
Please provide a resolution. Can you please check if there are any insecure uses of glibc or nghttp2 packages?
Cross posting this bug report here, at this might be the better place for it
Describe the bug
After upgrading fromconfluentinc/cp-ksqldb-server:6.1.0
toconfluentinc/cp-ksqldb-server:7.1.0
I've noticed warnings about ANTLR in the logs and after runningSHOW CONNECTORS
ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2 ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2
To Reproduce
Steps to reproduce the behavior, include:
We use Kubernetes for deployment with ArgoCD and a fork of the helm chart here.
- Update chart to use image tag
7.1.0
- Deploy
ksql> show connectors; ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2 ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2 ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2 ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2 Connector Name | Type | Class | Status ---------------------------------------- ---------------------------------------- ksql>
Expected behavior
No warnings. ANTLR versions should matchActual behaviour
A clear and concise description of what actually happens, including:
See aboveAdditional context
Log example[2022-04-05 21:23:46,591] INFO Adding function ComplexFunction for method public java.lang.Object io.confluent.ksql.function.UdfLoaderTest$ComplexUdf.foo(java.lang.String) (io.confluent.ksql.function.UdfLoader) ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2 ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2
This is a new image, so shouldn't be polluted with any changes we've made. This is a list off all the ANTLR JARs I found on the image. Only the bottom one is in the class path. I tried deleting it with
rm
, but that had no effect.['./usr/share/doc/cp-ksqldb-server/licenses/LICENSE-antlr4-runtime-4.7.1.txt', './usr/share/java/confluent-security/connect/antlr-runtime-3.5.2.jar', './usr/share/java/confluent-security/connect/antlr4-4.9.2.jar', './usr/share/java/confluent-security/connect/antlr4-runtime-4.9.2.jar', './usr/share/java/confluent-security/kafka-rest/antlr-runtime-3.5.2.jar', './usr/share/java/confluent-security/kafka-rest/antlr4-4.9.2.jar', './usr/share/java/confluent-security/kafka-rest/antlr4-runtime-4.9.2.jar', './usr/share/java/confluent-security/ksql/antlr-runtime-3.5.2.jar', './usr/share/java/confluent-security/ksql/antlr4-4.9.2.jar', './usr/share/java/confluent-security/ksql/antlr4-runtime-4.9.2.jar', './usr/share/java/confluent-security/schema-registry/antlr-runtime-3.5.2.jar', './usr/share/java/confluent-security/schema-registry/antlr4-4.9.2.jar', './usr/share/java/confluent-security/schema-registry/antlr4-runtime-4.9.2.jar', './usr/share/java/ksqldb-server/antlr4-runtime-4.7.1.jar']
When using the cp-ksqldb-server image I am getting this message written to stdout:
log4j:ERROR Could not instantiate class [org.apache.kafka.log4jappender.KafkaLog4jAppender].
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:190)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:304)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:123)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:755)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:738)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:652)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:518)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:577)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:504)
at org.apache.log4j.LogManager.(LogManager.java:119)
at org.slf4j.impl.Reload4jLoggerFactory.(Reload4jLoggerFactory.java:67)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
at io.confluent.admin.utils.cli.KafkaReadyCommand.(KafkaReadyCommand.java:53)
log4j:ERROR Could not instantiate appender named "kafka_appender".
SLF4J: A number (165) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
I do see kafka-log4j-appender-7.4.1-ccs.jar in the /usr/share/java/ksqldb-server directory.
This line and this line apparently make it impossible to run on k8s with readOnlyRootFilesystem: true
It would be better if the templates were in a different dir (perhaps a parent) so we can mount a writable dir at the target site; as-is, this forces a less-secure deployment.
I am thinking that instead of this
dub template "/etc/ksqldb/ksqldb-server.properties.template" "/etc/ksqldb/ksqldb-server.properties"
maybe something like this?
dub template "/etc/ksqldb/ksqldb-server.properties.template" "/etc/ksqldb/config/ksqldb-server.properties"
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.