zhumin-lv-wn / dinky

Dinky is an out of the box one-stop real-time computing platform dedicated to the construction and practice of Unified Streaming & Batch and Unified Data Lake & Data Warehouse. Based on Apache Flink, Dinky provides the ability to connect many big data frameworks including OLAP and Data Lake.

Dinky

Introduction

Dinky is an out-of-the-box, one-stop, real-time computing platform dedicated to the construction and practice of Unified Streaming & Batch and Unified Data Lake & Data Warehouse. Based on Apache Flink, Dinky provides the ability to connect many big data frameworks including OLAP and Data Lake.

Feature

Its main features are as follows:

• Immersive Flink SQL Data Development: Dinky provides automatic prompt completion, syntax highlighting, statement beautification, online debugging, syntax verification, execution plan, MetaStore, lineage, version comparison, and more.
• Support FlinkSQL multi-version development and execution modes: Dinky supports multiple development and execution modes for FlinkSQL, including Local, Standalone, Yarn/Kubernetes Session, Yarn Per-Job, and Yarn/Kubernetes Application.
• Support FlinkSQL syntax enhancement: Dinky enhances FlinkSQL with features like database synchronization, execution environments, global variables, statement merging, table-valued aggregate functions, load dependency, and row-level permissions.
• Supports real-time warehousing and lake entry of the entire FlinkCDC database, multi-database output, and automatic table creation.
• Support SQL job development: You can develop SQL jobs for various databases, including ClickHouse, Doris, Hive, MySQL, Oracle, Phoenix, PostgreSQL, Presto, SQL Server, StarRocks, and more.
• Support real-time online debugging:preview Table, ChangeLog, statistical chart and UDF.
• Support Flink Catalog, data source metadata online query and management.
• Support real-time task operation and maintenance: Online and offline, job information, cluster information, job snapshot, exception information, data map, data exploration, historical version, alarm record, etc.
• Support as multi-version FlinkSQL Server and OpenApi capability.
• Support real-time job alarm and alarm group: DingTalk, WeChat, Feishu, E-mail, etc.
• Support automatically managed SavePoint/CheckPoint recovery and triggering mechanisms: latest, earliest, specified, etc.
• Support resource management: Cluster instance, cluster configuration, jar, data source, alarm group, alarm instance, document, global variable, system configuration, etc.
• Support enterprise-level management: multi-tenant, user, role, project space.
• More hidden features await exploration by our users.

Principle

Run the Screenshot

FlinkSQL Studio

Grammar Check

Version Management

lineage

BI Charts

Metadata Query

Task Monitoring

Job Information

Participate in Contributions

Welcome to join the community, build a win-win situation, please refer to the contribution process： How to contribute.

Thank you to all the people who already contributed to Dinky!

How to Deploy

• The dev branch is the 0.8 reconstructed version, which is still unstable.

• The 0.7 branch is the current stable version. Please use this branch for compilation and debugging.

See source code compilation and installation and deployment for details.

How to Upgrade

Due to the numerous functionalities, there are several bugs and optimization points that need attention.. It is strongly recommended to use or upgrade to the latest version.

Upgrade steps:

(1) Upgrade the app: Replace all dependencies of latest Dinky.

(2) Upgrade the DDL: Execute the relevant version upgrade statements in the sql/upgrade directory, and execute them in the order of the version numbers.

Thanks

Standing on the shoulders of giants, Dinky was born. For this we express our heartfelt thanks to all the open source software used and its communities! We also hope that we are not only beneficiaries of open source, but also contributors to open source. We also hope that partners who share our enthusiasm and belief in open source will join us in contributing to the open-source community.

Below is a partial list of acknowledgements:

Apache Flink

FlinkCDC

Apache Flink Table Store

Apache Dolphinscheduler

Apache Doris

Druid

Ant-Design-Pro

Mybatis Plus

Monaco Editor

Sa Token

SpringBoot

Thanks to JetBrains for providing a free open-source license.

Get Help

1.Create an issue and provide a clear description.

2.Visit the official website website to read the latest documentation manual.

3.It is recommended to scan the code to enter the DingTalk group.

4.Enter the WeChat user community group (recommended, add WeChat wenmo_ai to invite into the group) and QQ user community group (543709668) to communicate, and apply for the remarks "Dinky + company name + position".

5.Follow the WeChat public account to get the latest official articles: Dinky Open Source.

6.Follow bilibili UP master (Shi Wen Mo A) to get the latest video teaching.

Copyright

Please refer to the LICENSE document.