microsoft / synckusto

Synchronize database schemas between a Kusto cluster and the local file system

Sync Kusto

The Sync Kusto tool was built to help create a maintainable development process around Kusto. The tool makes it easy to sync database schemas using either the local file system or a Kusto cluster as either the source of the target. It will copy full function definitions along with table schemas. Table data is not synced.

Scope

This tool is intended specifically for the developer desktop scenario. It will not help if you need to build an automated syncing process, generate change scripts, sync policies, etc. For all of that, we recommend using Delta Kusto.

Getting Started

Assuming that the user already has a Kusto database with functions and tables, set the Kusto database as the source and the local file system as the target. Press the compare button and the tool will find that none of the database objects are in the target. Press the update button to download everything from Kusto to the local file system. If it's the first time this tool has been run, the settings dialog will appear. Read through the settings below to get those configured correctly.

Recommended Team Process

The general usage pattern on our team is to make changes in a Kusto dev database which starts with a schema that mirrors the production database. Once the developer is satisfied with the changes, they use Sync Kusto with the development database as the source and their local file system as the target. The changes are committed to source control and a pull request is sent out for review. After the request is approved and the code merges to master, the developer syncs the master branch and uses Sync Kusto to sync from their local file system to the production Kusto database. This is a more flexible/manual version of the Azure Data Explorer task for Azure Dev Ops. We've had great success with this process and use the tool daily to help with our development and for deploying to production assets.

Sync Kusto Binaries

The release can be downloaded from the GitHub Releases page. Individual builds can also be found in the GitHub Actions page, or the repo can be cloned and built in Visual Studio.

Settings

Temporary Databases

When the local file system is selected as either the source or the target, Sync Kusto creates a temporary database containing all of the CSL files on the local file system. Specify a Kusto cluster and database where you have permissions to not only create new functions and tables but also to delete all the functions and tables that exist there already. This database will be completely wiped every time a comparison is run! If two users run a comparison pointing to the same temporary database at the same time, they'll get incorrect results. Ideally every user has their own temporary database. Note that you cannot specify a temporary database unless the database is empty. This is a safety check to avoid accidentally specifying a database that isn't intended to be wiped.

AAD Authority

If your username is in the form of UPN ([email protected]), you can hit the common AAD endpoint and your home tenant will be resolved automatically. If your tenant's automatic resolution does not work, you need to specify the AAD authority in the settings. Note that if you're planning to connect via an AAD application id and key then this setting is required regardless of tenant configuration.

Warnings

• Ask before dropping objects in the target Kusto database - This optional check is enabled by default and will prompt the user before dropping anything in the target database.

Formatting

• Place table fields on new lines - When CSL table files are written to a file target, each field is placed on a new line. This can make it easier to diff changes in a pull request.
• Generated ".create-merge table" commands instead of ".create table" - When CSL table files are written to a file target, the command will be ".create-merge table" if this is checked. If it is unchecked, the command will be ".create table".

Files

• Use legacy .csl file extensions - Kusto originally used files with .csl file extensions. This extension has since been deprecated and replaced with the .kql file extension. This setting tells synckusto which file extension should be used for reading and writing all files. The default state is checked, which means synckusto will consume and emit files with the .csl file extension.

Contributing

Issues, additional features, and tests are all welcome. See CONTRIBUTING.md for more information.

synckusto's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

External table definitions are not exported

Hello,

thank you for the tool! It really helps me out. However, I do not see external table definitions exported into csl files. Is this feature missing from the tool, or?

Thanks,
Davor

Generic connection error

Connection won't established between the Sync Kusto and the Kusto service.
Error window pointing there's missing user and password parameters though everything seems ok, and the Connection String against the Kusto Explorer works fine from the client station.

Database admin shouldn't be needed to sync from Kusto to File Path

Old versions of SyncKusto didn't require elevated Kusto permission to sync from Kusto to a local file path. A quick comparison of a functions in source control vs functions in Kusto shouldn't require any elevated permissions -- it's too risky

Database admin shouldn't be needed to sync from Kusto to File Path

Old versions of SyncKusto didn't require elevated Kusto permission to sync from Kusto to a local file path. A quick comparison of a functions in source control vs functions in Kusto shouldn't require any elevated permissions -- it's too risky

App should skip file when its filepath is too long

If one of the Source schema filepaths is too long, SyncKusto breaks with:
The Source schema is invalid: The folder path provided could not be found.

SyncKusto should ignore files whose paths are too long instead of breaking. The current behavior blocks syncing functions without the filepath issue.

BCDR - accidental data deletion

I ended up specifying my source database as temporary database as well without realizing the results. The tool cleanup up my source database without warnings. Now need to do some BCDR. Please add a warning when this happens or switch to another temporary database.

Publish in Microsoft Store

Could we publish this in the Microsoft Store to make installation/updating/etc easier/simpler?

Settings screen hangs after entering the details and upon clicking ok

The settings screen appears to hang when initially populating the temporary database cluster and database details.

symptoms are that the screen is not dismissed and the mouse pointer stay on hourglass mode and the app becomes unresponsive.

Eventually it needs closing via task manager.

Settings screen hangs when inputting settings

The settings screen appears to hang when initially populating the temporary database cluster and database details.

symptoms are that the screen is not dismissed and the mouse pointer stay on hourglass mode and the app becomes unresponsive.

Eventually it needs closing via task manager.

.kql instead of .csl?

The .csl extension is a vestige of the time when Kusto was called Cousteau. Should Sync Kusto move to using the .kql extension instead?