test-mass-forker-org-1 / delta-kusto

Engine able to compute delta between ADX clusters (and/or Kusto scripts) and generate update scripts

Delta-Kusto

Delta-Kusto is a Command-line interface (CLI) enabling Continuous Integration / Continuous Deployment (CI / CD) automation with Kusto objects (e.g. tables, functions, policies, security roles, etc.) in Azure Data Explorer (ADX) databases. It can work on a single database, multiple databases, or an entire cluster. It also supports multi-tenant scenarios.

Delta-Kusto is doing what SQL Database projects do for Microsoft SQL: enabling CI/CD, change management and source control of Kusto databases. It works with and produces Kusto scripts so it doesn't require a new language / serialization format and can therefore be used with other tools of the ADX ecosystem.

Delta-Kusto runs on Linux, Mac OS & Windows as a stand-alone executable. It is meant to be used in headless mode.

Delta-Kusto works on database structure, not data:

• Functions
• Tables / Columns
• Ingestion Mapping
• Policies
• Materialized Views
• Security roles
• External tables
• Continuous Export

Documentations

See the documentation for details and tutorial section for different tutorials on Delta-Kusto.

Overview

The high-level view of delta-kusto is the following:

The green boxes (current and target) represent sources. A source can be:

• ADX Database
• Kusto script

Delta-Kusto computes the delta between the two sources. The delta is a Kusto script containing the kusto commands required to take the current source and bring it to target source. The delta script can be exported as a stand alone file or as a folder hierarchy of scripts for easier readability and git-diff. It can also be applied to the current ADX Database.

Human validation often are required, especially if .drop commands are issued (to prevent unwanted data lost).

Scenarios / Flows

Using different combinations of sources can enable different scenarios:

Current	Target	Scenario	Description
ADX Database	Kusto scripts	CI / CD scenario	Push the state if a script set to an ADX database.
Kusto scripts	ADX Database	Determine gap between a script set and existing DB	Computed Delta script shows what would need to be added to the script set to obtain the state of the target database.
Empty	ADX Database	Reverse engineer a database	Special case of the previous scenario. The delta becomes the entire state of the target.
Kusto scripts	Kusto scripts	Offline sync	Compute a delta between two script sets. No live databases needed.
ADX Database	Kusto scripts	Live Sync	Find gap between two databases.

Limitations

The current release of Delta-Kusto includes functions, tables, ingestion mappings & the following policies:

• Auto Delete
• Caching
• Ingestion Batching
• Merge
• Retention
• Sharding
• Update

See the list of issues for details on upcoming features and bugs found.

Alternatives

• Sync Kusto - The Sync Kusto tool was built to help create a maintainable development process around Kusto.
• Azure DevOps Task for Azure Data Explorer - Azure Pipelines task that enables you to create release pipelines and deploy your database changes to your Azure Data Explorer databases.