leonardbang / flink-sql-etl Goto Github PK

hi Leonard:
问题1：
我把flink-connector-jdbc_2.11-1.11.1.jar、flink-connector-kafka_2.11-1.11.1.jar、mysql-connector-java-8.0.20.jar放到lib目录了，但是启动sql client报Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase，是不是需要把依赖的jar也拷贝到lib，如：flink-connector-kafka-base_2.11-1.11.1.jar、kakfa-clients_2.4.1.jar等等

问题2：
批计算，我是通过rest api定时提交批作业的。https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/monitoring/rest_api.html

但是如果我没有把flink-connector-jdbc_2.11-1.11.1.jar和mysql-connector-java-8.0.20.jar拷贝到lib目录的话，批作业任务提交多了就会导致Metaspace OOM，Metaspace调多大都没用

非常感谢

大咖，能否可以重新上传一个flink-demo-udf.jar，已有的flink-demo-udf.jar格式损坏了。
jar -vtf flink-demo-udf.jar
java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:225)
at java.util.zip.ZipFile.(ZipFile.java:155)
at java.util.zip.ZipFile.(ZipFile.java:126)
at sun.tools.jar.Main.list(Main.java:1115)
at sun.tools.jar.Main.run(Main.java:293)
at sun.tools.jar.Main.main(Main.java:1288)

hello Leonard：
1、CDC数据中如果存在一些非常规数据，比如较大的图片，文本，甚至有些图片能有上百兆，那么这类数据在kafka上传输时候，怎么处理其数据格式及其存储方式？

2、因为kafka本身定位不是用来持久化存储数据的，那么对于cdc数据的存放（下游sink可能存在复用或者数据订阅）是怎么处理的？

flink-sql-etl-master

org.example:data-generator >---------------------
[INFO] Building data-generator 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for io.confluent:kafka-avro-serializer:jar:5.3.0 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.505 s
[INFO] Finished at: 2021-03-17T10:40:49+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project data-generator: Could not resolve dependencies for project org.example:data-generator:jar:1.0-SNAPSHOT: Failure to find io.confluent:kafka-avro-serializer:jar:5.3.0