isabella232 / azure-data-explorer-go-sdk-example-to-ingest-data

Sample to demonstrate how to ingest (load) data into Azure Data Explorer using the Go SDK

page_type	languages	products	description	urlFragment
sample	go	azure	Sample to demonstrate how to ingest (load) data into Azure Data Explorer using the Go SDK	Azure-Data-Explorer-Go-SDK-example-to-ingest-data

Ingest data using the Azure Data Explorer Go SDK

Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. It provides a Go client library for interacting with the Azure Data Explorer service. You can use the Go SDK to query, control, and ingest into Azure Data Explorer clusters

You will start by using the sample program to create a table and data mapping in a test cluster. You then queue an ingestion to the cluster using the Go SDK and validate the results.

Getting Started

Prerequisites

• If you don't have an Azure subscription, create a free Azure account before you begin.
• You need Go installed on your computer. The Go SDK requires Go 1.13 as a minimum version
• A test cluster and database
• You need Git installed on your computer

Installation

To install using Go modules, create a module (if needed) and fetch the Azure Data Explorer package. For example:

go mod init foo.com/bar
go get github.com/Azure/azure-kusto-go/kusto

You should see the package dependency being added to the go.mod file

Azure Data Explorer resources

Create an Azure Data Explorer cluster and database before executing the sample program in this repository

Run the application

Clone the sample code:

git clone https://github.com/Azure-Samples/azure-data-explorer-go-ingest.git
cd azure-data-explorer-go-ingest

You will need to create Service Principal for authentication. For example, you can do it with Azure CLI using the az ad sp create-for-rbac command

Set Service Principal information along with cluster endpoint in the form of environment variables that will be used by the program:

export AZURE_SP_CLIENT_ID="<replace with appID>"
export AZURE_SP_CLIENT_SECRET="<replace with password>"
export AZURE_SP_TENANT_ID="<replace with tenant>"
export KUSTO_ENDPOINT="https://<cluster name>.<azure region>.kusto.windows.net"
export KUSTO_DB="name of the database"

Finally, to run the program:

go run main.go

You will see an output similar to the following:

Connected to Azure Data Explorer
Using database - testkustodb
Failed to drop StormEvents table. Maybe it does not exist?
Table StormEvents created in DB testkustodb
Mapping StormEvents_CSV_Mapping created
Ingested file from - https://kustosamplefiles.blob.core.windows.net/samplefiles/StormEvents.csv?st=2018-08-31T22%3A02%3A25Z&se=2020-09-01T22%3A02%3A00Z&sp=r&sv=2018-03-28&sr=b&sig=LQIbomcKI8Ooz425hWtjeq6d61uEaq21UVX7YrM61N4%3D

Resources

• Azure Data Explorer
• Azure Data Explorer Go SDK
• Azure Data Explorer Go SDK on GitHub
• Azure Data Explorer Go SDK reference