polatengin / indiana

Create Azure DevOps project items using a json template

Observability Epic for Engineering Projects

Overview

This script is designed to enhance the maturity of observability capabilities in our engineering projects. It allows users to add pre-defined epic, features, and user stories aligned with the Observability Maturity Model, facilitating the creation of a project backlog. The script supports three orchestrators - Azure DevOps (AzDo), GitHub, and Markdown - and can be customized to suit specific project needs.

The current list of work items is documented in doc/workitems.md.

Prerequisites

• Node.js and npm (Node Package Manager).
• Personal Access Token (PAT) for AzDo or GitHub, depending on the chosen orchestrator.

Run npm install to download all dependencies defined in package.json.

Running the Script

For Azure DevOps (AzDo) Orchestrator

ts-node index.ts --orchestrator azdo --token YOUR_AZDO_PAT --organization YOUR_ORG --project YOUR_PROJECT --file work_items.json

• --token: Your AzDo Personal Access Token.
• --organization: The name of your AzDo organization.
• --project: The name of your AzDo project.
• --file: The JSON file containing work items definitions.

For GitHub Orchestrator

ts-node index.ts --orchestrator github --token YOUR_GITHUB_PAT --organization YOUR_ORG --project YOUR_REPO --file work_items.json

• --token: Your GitHub Personal Access Token.
• --organization: The name of the user or organization owning the repo in GitHub.
• --project: The name of your GitHub repository.
• --file: The JSON file containing work items definitions.

Generating Documentation

To generate documentation after updating work_items.json (which contains the current work items), run the script with the Markdown orchestrator. This will produce Markdown in the target file (doc/workitems.md) with a structured documentation of the work items.

• --file: The JSON file containing work items definitions.
• --output: The output Markdown file where the work items will be documented.

The target file must already exist. The script will replace the contents of two blocks marked with start and end comments:

Table of Contents - <!-- toc --> and <!-- endtoc --> Work Items - <!-- workitems --> and <!-- endworkitems -->

Command:

ts-node index.ts --orchestrator markdown --file work_items.json --output doc/workitems.md

Ensure that work_items.json is updated with the latest work items before running the command. The script will parse the JSON file and convert the content into a structured Markdown document.

Notes

• Customize work items as needed but try to limit changes to Descriptions and Acceptance Criteria.
• For new additions or significant modifications, update them directly in the backlog after importing the Epic.
• The script is designed to be flexible and adaptable to different project requirements.

indiana's Issues

Select centralized monitoring solution (assumes Azure Monitor)

As a project lead, I want to make a decision on which centralized monitoring solution to use, so that telemetry data can be aggregated and analyzed

• Centralized monitoring solution selected

Design additional dimensions that should be added to existing logs

As a project developer, I want to define the business-specific dimensions that should be added to existing logs and traces, so that these attributes can be retrieved in KQL queries and support additional visualization tiles and alert rules

• Design doc created that lists additional required attributes/dimensions that are not available in the instrumented telemetry data

Auto instrument telemetry data

Instrumentation tooling should be added to the project to begin automatically instrumenting out-of-the-box logs, metrics, and traces

Design additional required business-specific metrics

As a project developer, I want to define the set of business-specific metrics that should be collected in addition to the out-of-the-box metrics, so that these new custom metrics can be implemented in code

• Design doc created that lists additional required metrics that are not instrumented by the application

Accommodate cost and scale

Cost and capacity concerns should be mitigated

Set retention policies for stored telemetry data

As a project developer, I want retention policies set for Log Analytics and Application Insights, so that data is archived after some time, reducing costs

• Log retention policy configured for Log Analytics and Application Insights in IaC

Set sampling percentage

As a project developer, I want sampling percentages set, so that a percentage of telemetry data is sampled out, reducing costs

• Sampling percentage configured for the application

Select instrumentation tooling (assumes OpenTelemetry exporter for Azure Monitor)

As a project lead, I want to make a decision on what tooling to use to instrument telemetry data, so that telemetry data can be collected and sent to the centralized monitoring solution

• Instrumentation library selected

Add log statements to project and configure log level

As a project developer, I want an application wide log level configured, and I want each existing log statement to have an appropriate level applied, so that application-wide log output becomes configurable

• Explanatory log statements added to project code, where missing
• Appropriate log level applied to all log statements
• Application wide log level configured (e.g. https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-7.0#configure-logging)

View and query resource metrics in Log Analytics

As a project developer, I want a set of basic KQL queries that utilize Log Analytics tables saved to the repository, so that the team can begin familiarizing with querying Log Analytics using KQL

• Documentation added that outlines where to retrieve resource logs and metrics
• At least 1 metric's retrieval process documented
• At least 3 KQL queries that use resource-specific Log Analytics Workspace tables documented (https://learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/tables-resourcetype)

Create Azure Workbooks resource

As a project developer, I want an Azure Workbook defined in IaC that includes the set of pre-built supporting queries, so that the visualization dashboard is deployed alongside existing project infrastructure

• Azure Workbooks defined in IaC that implements the dashboard outlined in the design doc using the defined KQL queries and metrics

Design cost and scaleability measures

As a project developer, I want clear specifications for cost and scalability measures, so that expenses related to the collection and storage of telemetry data are minimized

• Design doc created that defines the sampling percentage that should be used
• Log retention timeframe documented
• Log levels documented for all environments
• Processes that require log rotation identified and the rotation process documented

Implement advanced instrumentation practices

Additional telemetry data should be instrumented to support business-specific visualization tiles and alerts

Design tile layout

As a project developer, I want to predefine the set of tiles that should be displayed visually, so that project developers can build the necessary supporting queries

• Design doc created that outlines the list of telemetry signals that should be displayed in a dashboard
• Dashboard layout documented

Track custom dimensions in Azure Workbooks and Azure Alerts

As a project developer, I want to utilize the custom dimensions in Workbook tiles and Azure Alert rules, so that they can be visualized and trigger notifications when appropriate thresholds are hit

• Tiles added to Azure Workbook IaC that consume the custom dimensions
• Azure alert rules added to IaC that consume the custom dimensions

Ensure appropriate log levels are set across environments

As a project developer, I want log levels applied across all deployment environments, so that low level logs are suppressed by default, reducing costs

• Log level is set via environment variable
• Log level configured for all deployed environments
• Log level can be updated without re-building and re-deploying the application

Create infrastructure

All supporting observability infrastructure should be created using infrastructure-as-code

Plan observability approach

Observability related tooling should be selected and business critical indicators identified

Create Azure Alert rules

As a project developer, I want the set of pre-defined signals and thresholds added as Azure Alert rules to the project IaC, so that project admins receive alert notifications when certain thresholds are met

• Azure Alert rules defined in IaC that implement the rules outlined in the design doc using the defined KQL queries and metrics
• All alert rules utilize the action group

Ensure all supporting Azure resources send telemetry data to Log Analytics

As a project developer, I want all Azure resources to send diagnostic information to the Log Analytics Workspace, so that we can begin analyzing collected telemetry data for those resources

• Diagnostic settings added for all Azure resources in IaC
• Diagnostic settings reference Log Analytics Workspace

Create Log Analytics Workspace resource

As a project developer, I want a Log Analytics Workspace defined in IaC, so that we can send telemetry data for analysis

• Log Analytics Workspace defined in IaC

Set log rotation policies

As a project developer, I want log rotation policies applied where applicable, so that performance is unaffected while ensuring log data is properly migrated to Azure Monitor

• Log rotation processes implemented, where necessary

Identify critical business signals

As a project lead, I want to identify the resources and signals most important to the business and application, so that the team can begin collecting and analyzing the requisite data

• Design doc created that outlines telemetry signals to prioritize for the application
• Document references resource-specific and application-specific telemetry signals
• Monitor reference docs linked for each infrastructure resource (e.g. https://learn.microsoft.com/en-us/azure/service-bus-messaging/monitor-service-bus-reference)

Implement health checks

As a project developer, I want each service to expose a consumable health check, so that automated processes can access the check and report service health

• Health checks implemented on all services according to the type of service (http, background, etc.)
• Health check logs are queryable in Azure Monitor

Instrument new custom metrics

As a project developer, I want to instrument the additional business-specific custom metrics, so that these metrics can be captured in visualization tiles and alert rules

• Custom metrics added to application code to enable tracking of documented additional required metrics

Create Azure Alert action group

As a project developer, I want an Azure Alert action group with at least one email notification added to the project IaC, so that project admins can receive alert notifications

• Azure Alert action group defined in IaC

Reference Application Insights Instrumentation Key in project

As a project developer, I want the Application Insights Instrumentation Key to be automatically written to the project environment variables, so that telemetry data is sent to the Application Insights instance deployed via IaC

• Example environment variables file added to the project
• Environment variables file sets the Application Insights Instrumentation Key using the variable name specified by the OpenTelemetryExporter documentation
• Application Insights Instrumentation Key output from IaC as a sensitive value
• Application Insights Instrumentation Key output value automatically written to environment variables used by published versions of the project

Design alert signals and thresholds

As a project developer, I want to predefine the set of telemetry signals and corresponding thresholds that should trigger action, so that project developers can build the necessary supporting queries and IaC

• Design doc created that outlines the list of telemetry signals that should be used in alerts, with associated thresholds, etc
• Document references recommended alert rules for all supporting resources (e.g. https://learn.microsoft.com/en-us/azure/azure-monitor/vm/tutorial-monitor-vm-alert-recommended)

Track custom metrics in Azure Workbooks and Azure Alerts

As a project developer, I want to utilize the custom metrics in Workbook tiles and Azure Alert rules, so that they can be visualized and trigger notifications when appropriate thresholds are hit

• Tiles added to Azure Workbook IaC that consume the custom metrics
• Azure alert rules added to IaC that consume the custom metrics

Build supporting queries

As a project developer, I want the set of KQL queries that support the pre-defined visualization tiles added to the project, so that the queries can be added to dashboard related IaC

• Set of KQL queries that retrieve the specified log-based telemetry signals added to the existing dashboard design doc
• Metrics that are necessary to support dashboard tiles added to the document

Implement distributed tracing across service boundaries

As a project developer, I want distributed tracing configured across service boundaries, so that entire operations can be analyzed in Azure Monitor queries

• Appropriate trace and span IDs passed from service to service using W3C Trace Context (https://www.w3.org/TR/trace-context/)
• Application Insights Transaction Search displays entire end to end operation

Add auto instrumentation code to project

As a project developer, I want the language-specific OpenTelemetry Exporter for Azure Monitor to be added to the project, so that application-specific telemetry is collected

• Language-specific OpenTelemetry Exporter package installed (e.g. https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-enable?tabs=aspnetcore)
• Application code modified according to OpenTelemetry Exporter documentation

Create Application Insights resource

As a project developer, I want an Application Insights instance defined in IaC, so that we can send telemetry data for analysis

• Application Insights defined in IaC

Build visualization dashboards

Dashboards should provide visual representations of instrumented telemetry data

Add basic manual instrumentation

Manually instrumented telemetry data should augment auto-instrumented data to enable base visibility

Add custom dimensions to instrumented telemetry data

As a project developer, I want to instrument the additional business-specific custom dimensions, so that these new attributes can be captured in visualization tiles and alert rules

• Custom dimensions added to application code to enable tracking of documented additional required attributes/dimensions

Build automated alerts

Alerts should provide automatic notifications based on thresholds applied to incoming telemetry data

View and query logs, metrics, and traces in Application Insights

As a project developer, I want a set of basic KQL queries that utilize Application Insights tables saved to the repository, so that the team can begin familiarizing with querying Application Insights using KQL

• Documentation added that outlines how to retrieve application logs and metrics
• At least 1 metric's retrieval process documented
• At least 3 KQL queries that use resource-specific Application Insights tables documented (https://learn.microsoft.com/en-us/azure/azure-monitor/app/data-model-complete)

Ensure all dependencies are tracked

As a project developer, I want all currently untracked dependencies captured with spans, so that all sub-operations within the larger project operations are collected by Azure Monitor

• Requests and dependencies available in Application Insights compared with expected set of requests and dependencies for the application (many libraries will auto instrument spans for their requests and dependencies)
• Tracers and spans added to project code that wrap all missing requests and dependencies