Comments (10)
Hello @ysmintor, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.
你好 @ysmintor, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。
from dinky.
直接使用 kafka 连接器即可 本身都是 json
from dinky.
Is your requirement to split the data and write it to different tables?
from dinky.
Is your requirement to split the data and write it to different tables?
@aiwenmo Yes. One Kafka topic may have multiple cdc tables. And need to write into different tables. I also think we can conusme multile Kafka topics corresponding one table case.
from dinky.
I have actually take a look of Flink CDC and Hudi solutions. But it seems a bit hard to implement a connector from Kafka CDC (somethings I called it as debezium json in Kafka) to Hudi or other databases with my team group.
Recently I take some time to practice with Apache Paimon CDC ingestion of Kafka CDC, after that I thought it might a solution for us, as Apache Paimon serveral days ago became a Top Project of Apache graduated from incubation. So I wonder whether you can implement this Kafka CDC source connector or absorbe their implementation of KafkaSyncDatabaseAction and KafkaSyncTableAction or just wrap it into a CDCSOURCE task on Dinky.
I know these features may cause a bit code and structure changes, and please at your schedule to think that.
from dinky.
I have actually take a look of Flink CDC and Hudi solutions. But it seems a bit hard to implement a connector from Kafka CDC (somethings I called it as debezium json in Kafka) to Hudi or other databases with my team group.我实际上已经了解了 Flink CDC 和 Hudi 解决方案。但与我的团队一起实现从 Kafka CDC(我在 Kafka 中将其称为 Debezium json)到 Hudi 或其他数据库的连接器似乎有点困难。
Recently I take some time to practice with Apache Paimon CDC ingestion of Kafka CDC, after that I thought it might a solution for us, as Apache Paimon serveral days ago became a Top Project of Apache graduated from incubation. So I wonder whether you can implement this Kafka CDC source connector or absorbe their implementation of KafkaSyncDatabaseAction 和 KafkaSyncTableAction or just wrap it into a CDCSOURCE task on Dinky.最近我花了一些时间练习 Apache Paimon CDC 对 Kafka CDC 的摄取,之后我认为这可能是我们的一个解决方案,因为 Apache Paimon 几天前已经成为 Apache 孵化的顶级项目。所以我想知道您是否可以实现这个 Kafka CDC 源连接器或吸收他们的 KafkaSyncDatabaseAction 和 KafkaSyncTableAction 实现,或者只是将其包装到 Dinky 上的 CDCSOURCE 任务中。
I know these features may cause a bit code and structure changes, and please at your schedule to think that.我知道这些功能可能会导致代码和结构发生一些变化,请在您的日程安排中考虑这一点。
Do you have the energy to fulfill this requirement?
from dinky.
@aiwenmo @Zzm0809
I have actually take a look of Flink CDC and Hudi solutions. But it seems a bit hard to implement a connector from Kafka CDC (somethings I called it as debezium json in Kafka) to Hudi or other databases with my team group.我实际上已经了解了 Flink CDC 和 Hudi 解决方案。但与我的团队一起实现从 Kafka CDC(我在 Kafka 中将其称为 Debezium json)到 Hudi 或其他数据库的连接器似乎有点困难。
Recently I take some time to practice with Apache Paimon CDC ingestion of Kafka CDC, after that I thought it might a solution for us, as Apache Paimon serveral days ago became a Top Project of Apache graduated from incubation. So I wonder whether you can implement this Kafka CDC source connector or absorbe their implementation of KafkaSyncDatabaseAction 和 KafkaSyncTableAction or just wrap it into a CDCSOURCE task on Dinky.最近我花了一些时间练习 Apache Paimon CDC 对 Kafka CDC 的摄取,之后我认为这可能是我们的一个解决方案,因为 Apache Paimon 几天前已经成为 Apache 孵化的顶级项目。所以我想知道您是否可以实现这个 Kafka CDC 源连接器或吸收他们的 KafkaSyncDatabaseAction 和 KafkaSyncTableAction 实现,或者只是将其包装到 Dinky 上的 CDCSOURCE 任务中。
I know these features may cause a bit code and structure changes, and please at your schedule to think that.我知道这些功能可能会导致代码和结构发生一些变化,请在您的日程安排中考虑这一点。Do you have the energy to fulfill this requirement?
Sorry, I do not have resources to implement this feature.
from dinky.
I am willing to submit a PR.
from dinky.
@aiwenmo @Zzm0809
I have actually take a look of Flink CDC and Hudi solutions. But it seems a bit hard to implement a connector from Kafka CDC (somethings I called it as debezium json in Kafka) to Hudi or other databases with my team group.我实际上已经了解了 Flink CDC 和 Hudi 解决方案。但与我的团队一起实现从 Kafka CDC(我在 Kafka 中将其称为 Debezium json)到 Hudi 或其他数据库的连接器似乎有点困难。
Recently I take some time to practice with Apache Paimon CDC ingestion of Kafka CDC, after that I thought it might a solution for us, as Apache Paimon serveral days ago became a Top Project of Apache graduated from incubation. So I wonder whether you can implement this Kafka CDC source connector or absorbe their implementation of KafkaSyncDatabaseAction 和 KafkaSyncTableAction or just wrap it into a CDCSOURCE task on Dinky.最近我花了一些时间练习 Apache Paimon CDC 对 Kafka CDC 的摄取,之后我认为这可能是我们的一个解决方案,因为 Apache Paimon 几天前已经成为 Apache 孵化的顶级项目。所以我想知道您是否可以实现这个 Kafka CDC 源连接器或吸收他们的 KafkaSyncDatabaseAction 和 KafkaSyncTableAction 实现,或者只是将其包装到 Dinky 上的 CDCSOURCE 任务中。
I know these features may cause a bit code and structure changes, and please at your schedule to think that.我知道这些功能可能会导致代码和结构发生一些变化,请在您的日程安排中考虑这一点。Do you have the energy to fulfill this requirement?
Sorry, I do not have resources to implement this feature.
只需要使用dinky flink jar的方法调用 paimon action 来做kafka cdc, 无需做单独的source的;但目前的问题事paimon 0.8的版本依然没解决debezium json的自动寻找主键的问题,debezium json的格式是不带主键信息的,无法识别主键,paimon无法自动建表,从paimon的issue看,有人实现了从kafka connect 里生成的key去取主键,暂时没有merge到master的branch里,需要等待0.9版本了;但目前dinky的datastream-kafka生成到kafka信息只有value,没有key,这也不会被paimon cdc所识别
from dinky.
Hello @, this issue has not been active for more than 30 days. This issue will be closed in 7 days if there is no response. If you have any questions, you can comment and reply.
你好 @, 这个 issue 30 天内没有活跃,7 天后将关闭,如需回复,可以评论回复。
from dinky.
Related Issues (20)
- [Bug] [develop module] yarn-application Submit failed
- [Bug] Git project page refreshes infinitely
- [Bug] [CDCSOURCE] MysqlCDC to paimon with hive catalog error HOT 4
- [Feature][studio] Support FlinkSQL Procedures
- [Bug] [Module Name] Bug title After deploying dinky-release-1.19-1.1.0.tar.gz for the first time, there is no data. When I click on the workbench, an exception is reported. HOT 2
- [Bug] [web] The enable button in the form does not have a default value when creating a new Git project HOT 1
- [Bug] [admin] Type mismatch caused by tenantId field in git project entity class when creating a new git project item HOT 2
- [Bug] [admin] Cannot create a new git project item after deleting all list items
- dinky-app-1.19-1.1.0-jar-with-dependencies.jar path error HOT 1
- [Optimization][Data Studio] Recommend display the oracle schema in pagination HOT 1
- [Bug] [Data Development] When submitting a task, I found that there is no application cluster to choose from. HOT 1
- [Optimization] [Module Name] Failed to submit task using yarn-application HOT 3
- [Bug] The dispatcher returned from the ServletContext does not support asynchronous dispatching HOT 15
- [Bug] Caused by: java.lang.IllegalArgumentException: EndPoint is not set HOT 10
- [Feature][*] Release 1.1.1 HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli is reported when starting yarn-application in the hadoop cluster /Option$Builder;flink version is 1.18, hadoop cluster is version 3.3.6 HOT 7
- [Optimization][web] When there are too many tables in a database of the data source, the tree list of data source details expands and gets stuck. HOT 1
- During oracle cdc synchronization, the databasename and sid of the oracle source database are different, resulting in the inability to connect to the database. HOT 5
- AggTable is faulty in this document HOT 2
- [Document][doc] Update ICP
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dinky.