azure / azure-kusto-trender

A JavaScript client for Kusto (Azure Data Explorer, Synapse Data Explorer, Kusto free), featuring components for making calls directly to Kusto, rendering results from the API, and more

KustoTrender: The Azure Data Explorer JavaScript SDK

The KustoTrender JavaScript SDK is a JavaScript library for Microsoft Azure Data Explorer, featuring components for data visualization and analytics, utilities for making calls directly to the KQL Cluster, and more. KustoTrender also ships with an associated CSS file (which you must include using your preferred css linking method), which makes the components look great out of the box.

Resources

• API Reference documentation
• Product documentation
• Hosted KustoTrender samples

Installing

If you use npm, npm install azure-kusto-trender.

To import all of KustoTrender into an ES2015 application, import everything into a namespace, like so:

import KustoTrender from "azure-kusto-trender";

// later, when you want a line chart
let kustoTrender = new KustoTrender();
let lineChart = new kustoTrender.ux.LineChart(document.getElementById('chart'));

You can also import components individually. If you only need the LineChart, you can import it like so...

import LineChart from 'azure-kusto-trender/LineChart'

// later when you want a line chart
let lineChart = new LineChart(document.getElementById('chart'));

Importing individual components can help significantly reduce your bundle size as they work better with tree shaking. This is the recommended approach if your app only consumes specific components.

To import the kustotrender stylesheet into an ES2015 application, import either kustotrender.css or kustotrender.min.css, like so...

import 'azure-kusto-trender/kustotrender.css' // Standard styles
import 'azure-kusto-trender/kustotrender.min.css' // Minified styles

How to run Trender on your Kusto

A Kusto-Trender experience (Example 1, Example 2) relies on the following fundamental items:

• An Azure Active Directory App
• A Kusto cluster
• Schema & stored functions deployed on the cluster
• Optional: Sample data / data pipeline

This section covers the steps you need to take in order to run the Kusto-Trender on your Kusto environment.

1. Create an Azure Application
2. Create a Kusto environment
3. Create the schema
4. Optional: Deploy sample data
5. Optional: Adapt to your data

Create an Azure Application

If your Trender needs to access Kusto using the credentials of the calling user, configure delegated permissions for your application registration. Please follow the following documentation:

1. Create an Azure Active Directory application. It's not needed to create any certificate or secret.
2. Configure delegated permissions for the application registration.
3. Optional: grant admin consent for the permissions of the app you created.

Create a Kusto environment

Kusto is available in many incarnations nowadays:

• Azure Data Explorer
• Azure Synapse Data Explorer
• MyFreeCluster

All of them are compatible with Kusto Trender. The one that does not come with any cost is MyFreeCluster. You'll be able to create a new cluster for you in seconds.

Create the schema

Once you have a working Kusto, you want to deploy the schema. You can find the schema that fuels the examples here. It's a small script which creates a couple of tables and functions. Just execute it in the scope of the database you like to use.

The following tables are created by the script.

Timeseries

The raw timeseries data. The TimeseriesId column represents an unique identifier for a timeseries (some call it "tag").

Timeseries
| take 3

TimeseriesId	Timestamp	Value
939d52f1-cba7-48bb-87cb-d8e5d9050a73	2017-05-01 00:00:00.0000000	0,321766886722178
011523e8-5ab6-46ec-9a50-4ac3a6488f8c	2017-05-01 00:00:00.0000000	0,00480370965876755
bddbab16-fab5-4d59-8353-f35b58866e53	2017-05-01 00:00:00.0000000	0,84581159990962

TimeseriesMetadata

The metadata for the timeseries including name, description and other columns.

TimeseriesMetadata
| take 3

TimeseriesId	TimeseriesName	Description	EngUnits	DataStreamTags
9fe3d2cc-6395-4f01-943a-64bcfafc2f90	vibration-1	A vibration sensor	Hz	[]
e8f32758-ad81-4471-8e4d-62eefa74f314	vibration-2	A vibration sensor	Hz	[]
34994e72-205e-4288-a0e4-5b0a615f2c0d	vibration-3	A vibration sensor	Hz	[]

TimeseriesHierarchy

The hierarchy of the timeseries. In many scenarios the timeseries (aka tags) are organized in a hierarchy. The functions created in the last step allow browsing of that hierarchy. The Path is created as an array of path elements. The top level element is considered to be the name of the hierarchy. The remaining elements are considered to be the structural vertices of the hierarchy.

TimeseriesHierarchy
| take 3

TimeseriesId	DisplayName	Path
687f97ef-268e-4daf-932c-181f793d0e59	vibration-10	[  "Contoso Plant 1",  "Factory1",  "Line1",  "Station3" ]
e1168559-15d5-4fc7-bf2c-4b773e4f8d1b	temp control rate-42	[  "Contoso Plant 1",  "Factory1",  "Line1",  "Station3" ]
283eebcb-d267-4a76-8179-a9b33fb87cf2	temperature-16	[  "Contoso Plant 1",  "Factory1",  "Line1",  "Station3" ]

Deploy sample data

In order to deploy the sample data you need to execute the script.

Adapt to your data / schema

This section is relevant if you collect data and store it in a different schema then they one supported by the sample scenarios. There are multiple ways on how to proceed.

• You can create a lightweight ETL pipeline inside of Kusto using Update Policies.
• A full blown ETL process before you ingest data into Kusto. There are tons of tools available.
• Instead of rolling out the schema in the samples you could create views (stored functions without parameters) in Kusto which mimic the schema. The functions of the schema ddl file would still work.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

azure-kusto-trender's Issues

SearchTagsAtPath not working

In ADXTrenderClient there's a function searchTagsAtPath which does a query to ADX with a function called SearchTagsAtPath_Henning, which is not found in the trender-schema.kql which is supposed to create all the necessary functions in the ADX. It seems to me that I can just redirect those calls to getChildrenTags function in ADXTrenderClient instead, which is identical except for the function it uses in ADX. Why are these two in separate logic paths in the HierarchyDelegate? More over there's identical "else if" branches there that call the searchTagsAtPath.

Error: Command is not enabled in the current deployment mode.

Error: Command is not enabled in the current deployment mode.

File: https://github.com/Azure/azure-kusto-trender/blob/master/kusto/trender-schema.kql
Line 11 is not compatible with ADX Free Clusters.

.alter tables (Timeseries, TimeseriesHierarchy, TimeseriesMetadata) policy caching hot = 3650d
CommandType: CachingPolicyAlter
Result: Failed
Reason: Command is not enabled in the current deployment mode.

Breaks schema deploy midway and UDF functions do not get created.

Recommend commenting out line 11 till error handling can be implemented for free clusters.

Error running basic script with node

Running this code:

// @ts-ignore
import KustoTrender from 'azure-kusto-trender';

const kustoTrender = new KustoTrender();

console.log(kustoTrender);

produces the following error:

ReferenceError: self is not defined
    at Object.<anonymous> (C:\...\kusto-test\node_modules\azure-kusto-trender\kustotrender.js:2:224)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:169:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:530:24)
    at async loadESM (node:internal/process/esm_loader:91:5)

Technical info

• Azure kusto trender 0.0.9

Mismatch in package name

On the npm registry the package is named azure-kusto-trender, however in the package.json and docs the package is named kustotrender.

Typescript error when importing `azure-kusto-trender`

When importing azure-kusto-trender the following error is reported by typescript:

src/index.ts:1:26 - error TS7016: Could not find a declaration file for module 'azure-kusto-trender'. 'C:/Users/steeuwisse.ext/Desktop/kusto-test/node_modules/azure-kusto-trender/kustoTrender.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/azure-kusto-trender` if it exists or add a new declaration (.d.ts) file containing `declare module 'azure-kusto-trender';`

1 import KustoTrender from 'azure-kusto-trender';

Notes
Might be related to #37

Docs: connecting to azure data explorer

I have a few questions regarding to the code about connecting to data explorer.

• Is it correct that this piece of code provide a connection to data explorer?
• What do the three parameters mean?
• How and where can the parameter values be found for the user?
• Does TSItoken stand for time series insight token and if so, why is this referred to time series insight?

const cluster = new ADX.ADXTrenderClient( "https://help.kusto.windows.net/", authContext.getTsiToken, "Trender" );

`

When will this be uploaded?

Hello!

Quite interested in this, is it going to be uploaded?

Azure Kusto Trender - Integration with Angular Application

Hi all,
I'm trying to integrate this library into a plain angular application, but I'm not able to make it work.
These are the steps I've used to setup the sample solution:

1. Created a new angular application using the following command:

ng new my-kusto-app

2. Added the package azure-kusto-trender:

cd ./my-kusto-app
npm install azure-kusto-trender

3. Modified the app.component.html as:

<h3>{{title}}</h3>

<div id="chart1"></div>

<button (click)="LoadLineChart()">Load Chart</button>

<div>{{status}}</div>

4. Included the kustotrender.js in the angular.json file, modifing the script section:

"scripts": [
  {
    "input": "./node_modules/azure-kusto-trender/kustoTrender.js",
    "inject": false,
    "bundleName": "kustoTrender"
  }
]

5. Modified the index.html adding the <script> section in the header in the header

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>MyKustoApp</title>
  <base href="/">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <link rel="icon" type="image/x-icon" href="favicon.ico">
  <script src="kustoTrender.js"></script>
</head>
<body>
  <app-root></app-root>
</body>
</html>

6. Implemented the LoadLineChart function in the app.component.ts file as:

public LoadLineChart(){
    var data = [];

    var minute: number = 60 * 1000;
    var n: number = new Date().valueOf();
    n = Math.floor(n / minute) * minute;

    var from = new Date(n);

    var to: Date;

    for (var i = 0; i < 3; i++) {
      var lines: any = {};
      data.push({ [`Factory${i}`]: lines });
      for (var j = 0; j < 5; j++) {
        var values: any = {};
        lines[`Station${j}`] = values;
        for (var k = 0; k < 60; k++) {
          if (!(k % 2 && k % 3)) {  // if check is to create some sparseness in the data
            var to = new Date(from.valueOf() + 1000 * 60 * k);
            var val = Math.random();
            var val2 = Math.random();
            var val3 = Math.random();
            var val4 = Math.random();
            values[to.toISOString()] = { avg: val, x: val2, y: val3, r: val4 };
          }
        }
      }
    }

    var swimLaneOptions = {
      1: { yExtent: [0.2, 0.8], yAxisType: 'shared', label: 'Lane 1', onClick: () => alert("'Lane 1' label/axis clicked") },
      2: { yAxisType: 'shared', label: 'Lane 2', onClick: () => alert("'Lane 2' label/axis clicked") },
      3: { yAxisType: 'shared', label: 'Lane 3', onClick: () => alert("'Lane 3' label/axis clicked") }
    };

    try {
      var kustoTrender = new KustoTrender();

      var lineChart = kustoTrender.ux.LineChart(document.getElementById('chart1'));

      lineChart.render(data, { theme: 'light', swimLaneOptions });
    }
    catch (err: any) {
      this.status = err. Message;
    }
  }

The angular app returns the following erroe:
lineChart.render is not a function

Using the Edge DevTools, it seems that the lineChart object seems to be instantiated, but the render method is not there:

Please, could you help me with this? Is this a problem with the packages? Any way to solve it?

Errors when running tsc

When running tsc on a clone of the 0.0.9 release, the following errors are produced:

node_modules/@types/awesomplete/index.d.ts:12:30 - error TS2304: Cannot find name 'NodeSelector'.

12     static $$(expr: string | NodeSelector, con?: any): NodeList;
                                ~~~~~~~~~~~~

node_modules/@types/awesomplete/index.d.ts:15:38 - error TS2304: Cannot find name 'NodeSelector'.

15         (expr: string|Element, con?: NodeSelector): string | Element;
                                        ~~~~~~~~~~~~

node_modules/@types/d3-shape/index.d.ts:2273:19 - error TS2304: Cannot find name 'CanvasPathMethods'.

2273     draw(context: CanvasPathMethods, size: number): void;
                       ~~~~~~~~~~~~~~~~~

node_modules/@types/eslint/index.d.ts:450:42 - error TS2724: '"C:/Users/steeuwisse.ext/Desktop/azure-kusto-trender/node_modules/@types/estree/index"' has no exported member named 'ChainExpression'. Did you mean 'ThisExpression'?

450         ChainExpression?: ((node: ESTree.ChainExpression & NodeParentExtension) => void) | undefined;
                                             ~~~~~~~~~~~~~~~

node_modules/@types/eslint/index.d.ts:473:43 - error TS2694: Namespace '"C:/Users/steeuwisse.ext/Desktop/azure-kusto-trender/node_modules/@types/estree/index"' has no exported member 'ImportExpression'.

473         ImportExpression?: ((node: ESTree.ImportExpression & NodeParentExtension) => void) | undefined;
                                              ~~~~~~~~~~~~~~~~

src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1694:23 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(selector: string): Selection<BaseType, unknown, HTMLElement, any>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'string'.
  Overload 2 of 2, '(node: BaseType): Selection<BaseType, unknown, null, undefined>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'BaseType'.
      Type 'ParentNode' is missing the following properties from type 'Document': URL, alinkColor, all, anchors, and 188 more.

1694             d3.select(this.parentNode).select(".tsi-filter-icon").node()
                           ~~~~~~~~~~~~~~~


src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1698:25 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(selector: string): Selection<BaseType, unknown, HTMLElement, any>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'string'.
  Overload 2 of 2, '(node: BaseType): Selection<BaseType, unknown, null, undefined>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'BaseType'.

1698                 .select(this.parentNode)
                             ~~~~~~~~~~~~~~~


src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1709:23 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(selector: string): Selection<BaseType, unknown, HTMLElement, any>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'string'.
  Overload 2 of 2, '(node: BaseType): Selection<BaseType, unknown, null, undefined>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'BaseType'.

1709             d3.select(this.parentNode).select(".tsi-filter-icon").node()
                           ~~~~~~~~~~~~~~~


src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1713:25 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(selector: string): Selection<BaseType, unknown, HTMLElement, any>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'string'.
  Overload 2 of 2, '(node: BaseType): Selection<BaseType, unknown, null, undefined>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'BaseType'.

1713                 .select(this.parentNode)
                             ~~~~~~~~~~~~~~~


src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1793:23 - error TS2769: No overload matches this call.
  Overload 1 of 2, '(selector: string): Selection<BaseType, unknown, HTMLElement, any>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'string'.
  Overload 2 of 2, '(node: BaseType): Selection<BaseType, unknown, null, undefined>', gave the following error.
    Argument of type 'ParentNode' is not assignable to parameter of type 'BaseType'.

1793             d3.select((this as HTMLElement).parentNode)
                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



Found 10 errors in 4 files.

Errors  Files
     2  node_modules/@types/awesomplete/index.d.ts:12
     1  node_modules/@types/d3-shape/index.d.ts:2273
     2  node_modules/@types/eslint/index.d.ts:450
     5  src/UXClient/Components/HierarchyNavigation/HierarchyNavigation.ts:1694

Notes
Deps were installed with npm i --legacy-peer-deps due to #36, could be related

Order in Linechart / applying of chartDataOptions

Hello all,
I have a problem with the sorting of the individual curves in the lineChart.
As soon as several curves are added, their order is set arbitrarily. (Neither alphabetically, not the order in which they were added, nor do they keep the order from the hierarchy tree).
This problem occurs not only in my own implementation but also in the Fully featured trender from the example page.
On its own, this is not a major issue, but I suspect it is related to a more serious problem.
As soon as I want to adjust chartDataOptions like showDots or yExtent via the contextMenu for only one curve "A" (and not for all curves), this change is applied to another curve "B".

Technical Info:

• azure-kusto-trender 0.0.9

npm install fails

Running npm install on a clone of the 0.0.9 release fails with the following error:

npm ERR! code ERESOLVE
npm ERR! ERESOLVE could not resolve
npm ERR!
npm ERR! While resolving: [email protected]
npm ERR! Found: [email protected]
npm ERR! node_modules/webpack
npm ERR!   dev webpack@"^5.72.0" from the root project
npm ERR!   peer webpack@"4.x.x || 5.x.x" from @webpack-cli/[email protected]
npm ERR!   node_modules/@webpack-cli/configtest
npm ERR!     @webpack-cli/configtest@"^1.1.1" from [email protected]
npm ERR!     node_modules/webpack-cli
npm ERR!       dev webpack-cli@"^4.9.2" from the root project
npm ERR!       3 more (@webpack-cli/configtest, @webpack-cli/info, @webpack-cli/serve)
npm ERR!   6 more (css-loader, terser-webpack-plugin, webpack-cli, ...)
npm ERR!
npm ERR! Could not resolve dependency:
npm ERR! peer webpack@"^2.0.0 || ^3.0.0 || ^4.0.0" from [email protected]
npm ERR! node_modules/file-loader
npm ERR!   dev file-loader@"^1.1.11" from the root project
npm ERR!
npm ERR! Conflicting peer dependency: [email protected]
npm ERR! node_modules/webpack
npm ERR!   peer webpack@"^2.0.0 || ^3.0.0 || ^4.0.0" from [email protected]
npm ERR!   node_modules/file-loader
npm ERR!     dev file-loader@"^1.1.11" from the root project
npm ERR!
npm ERR! Fix the upstream dependency conflict, or retry
npm ERR! this command with --force, or --legacy-peer-deps
npm ERR! to accept an incorrect (and potentially broken) dependency resolution.

eresolve-report.txt

Technical info:

• Node v18.12.1
• npm 8.12.1
• Microsoft Windows 10 Enterprise (10.0.19044 Build 19044)

Notes
Running npm i --legacy-peer-deps seems to work, albeit with 21 vulnerabilities (6 moderate, 13 high, 2 critical)

Data sample storage account not accessible

Hello,

All the data ingestion commands in the kusto/trender-sample-data.kql script failed with this error message:

"Operation": DataIngestPull,
"State": Failed,
"Status": Ingestion failed for all sources. Failure example reason: Failed to download blob: Access to persistent storage path 'https://trendersampledata.blob.core.windows.net/data/TimeseriesHierarchy_1_814905e48a164222926254296d280149.csv.gz' was denied (operation 'BlobPersistentStorageFile2.DownloadToFileAsync'),

When I try to access it directly to the web browser, it say : "AccoundIsDisable"

AccountIsDisabled
The specified account is disabled. RequestId:93cde93e-801e-0019-5aa2-ebb1b2000000 Time:2023-09-20T09:10:57.9440090Z

Thanks,
Antoine