azure / azure-sdk

This is the Azure SDK parent repository and mostly contains documentation around guidelines and policies as well as the releases for the various languages supported by the Azure SDK.

Azure SDK

The Azure SDK delivers a platform for developers to leverage the wide variety of Azure services in their language of choice. The source for the client libraries exists for the most part in repositories for each language. This repository is meant to be a jumping off point into those language specific repositories. Issues related to a specific language should be opened in the corresponding repository but cross cutting issues can be opened in this repository.

Language	Design Guidelines	Packages	Repo	Documentation
General	General Design Guidelines		azure-sdk Repository	Official Azure Documentation
Android	Design Guidelines for Android (Draft)	Android Packages	azure-sdk-for-android Repository	Coming Soon
C# /.NET	Design Guidelines for .NET	.NET Packages	azure-sdk-for-net Repository	.NET Documentation
Go	Design Guidelines for Go (Draft)	Go Packages	azure-sdk-for-go Repository	Go Documentation
C	Design Guidelines for C99 (Draft)	C Packages	azure-sdk-for-c Repository	C Documentation
C++	Design Guidelines for C++ (Draft)	C++ Packages	azure-sdk-for-cpp Repository	C++ Documentation
iOS	Design Guidelines for iOS (Draft)	iOS Packages	azure-sdk-for-ios Repository	Coming Soon
Java	Design Guidelines for Java	Java Packages	azure-sdk-for-java Repository	Java Documentation
JavaScript	Design Guidelines for TypeScript	JavaScript Packages	azure-sdk-for-js Repository	JavaScript Documentation
Python	Design Guidelines for Python	Python Packages	azure-sdk-for-python Repository	Python Documentation

Service teams should schedule reviews of their client libraries with the Azure SDK Architecture Board. See the Review Process for more information.

Terminology

• SDK: Software Development Kit. This refers to the entire Azure SDK for a single language, itself broken up into numerous Azure SDK Client Libraries (as defined below).

• Client Library. This refers to a library (and associated tools, documentation, and samples) that customers/developers use to ease working with an Azure service. There is often one client library per service and per programming language. Sometimes a single client library will contain the ability to connect to multiple services. Each client library is published separately to the appropriate language-specific package repository. These releases are performed exclusively by the Azure SDK engineering systems team. Customers/Developers consume and use each client library separately as necessary to solve their use case.

• Package. This refers to a client library after it has been packaged for distribution for customer-developers to consume. Examples are:

  • A NuGet package for a .NET client library
  • A Maven package for a Java library
  • An npm package for a JavaScript library
  • A Python wheel for a Python library

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

Please see our contributing guide for complete instructions on how you can contribute to the Azure SDK. For instructions on how to contribute to each individual SDK, please check the CONTRIBUTING.md in each language repo.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Internal azure-sdk DevOps Wiki

azure-sdk's Issues

Investigate Pipelines failures building PRs from private forks

It appears that Azure Pipelines cannot clone sources of PRs from forks. Might be specific to private repos and the "DevOps GitHub App" for auth.
Examples:

https://dev.azure.com/azure-sdk/public-staging/_build/results?buildId=540&view=logs
https://dev.azure.com/azure-sdk/public-staging/_build/results?buildId=538&view=logs

Error:

fatal: could not read Username for 'https://github.com': terminal prompts disabled

We should get a minimal repro and report to the Azure Pipelines team.

Expected outcomes

We either need to:

1. [Varify] Be able to create forks of private repos and validate PRs from the forks in the CI for the private repo.
2. [Document] Provide documentation clarifying that private forks of private repos will not work.

Package templates exist for each language

• Azure DevOps Organization
  • https://dev.azure.com/azure-sdk/
• Languages
  • .NET
  • Java
  • Python
  • JavaScript
• In each language
  • One class library project which generates a reusable class library (NuGet Package, Maven Package, etc)
  • One test project which verifies class library
  • VSTS pipeline to restore, build, run tests, publish package
  • Documentation for developer to manually build, run tests, etc

CC: @weshaggard

Design Guidelines: Customization of Networking Stack

Write design guidelines for customizing networking stack in Azure SDKs, a.k.a. the "pipeline"

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python Guidelines

Move private repos from DevOps to GitHub

Tasks

• Create private GitHub repos
• Mirror/Authorize
  • Mirror private GitHub repos to private DevOps repos (https://github.com/dotnet/arcade/blob/master/Documentation/VSTS/VSTSOnboarding.md)
    OR
  • Authorize private DevOps pipeline to build private GitHub repo
• Import code (https://help.github.com/articles/importing-a-git-repository-using-the-command-line/)
  • dotnet
  • java
  • python

Create Team for DevOps knowledge sharing

Create a Teams "team" for DevOps knowledge sharing with IoT, .NET Core

Core Concept: Region

Design Guidelines: Supporting Cancellations and Timeouts

Write design guidelines for supporting cancellations and timeouts in Azure SDKs

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python Guidelines

Test packages in both live and simulated environments

All tests should be runnable against both live Azure and some form of locally-runnable simulated environment. This simulated environment could be the actual service code running in a container (e.g. CosmosDB), or a set of recorded HTTP requests and responses and scripts for how they should be replayed.

Ideally we would have a single simulator per service which can be used for tests in all languages. We do not want a different simulator per service per language, both because it's duplicated work and could lead to inconsistencies.

CC: @weshaggard

Design Guidelines: Basic API Structure

Write design guidelines for basic API structure of Azure SDKs, i.e. main types, naming conventions, etc.

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python Guidelines

Add a few existing SDKs to engineering system

• SDKs
  • KeyVault
  • Batch
• Languages
  • .NET
  • Java
  • Python
  • JavaScript
• Steps
  • Restore
  • Build
  • Test
  • Create package
  • Publish package
• Build types
  • Nightly / per-checkin
    • Signing?
    • Publish to MyGet (or equivalent)
  • PR Validation
    • No publish
    • No signing
    • Community PR (untrusted) vs Own PR (trusted)
  • Official
    • Signing
    • Publish to NuGet (or equivalent)
• Test types
  • Live Azure
  • Simulated service

Move PR triggers from designer into yaml

Our PR triggers should be moved from the designer into yaml, once they are supported (microsoft/azure-pipelines-yaml#38).

According to the docs, this is now supported.

Repos

• azure-sdk-for-net
• azure-sdk-for-java
• azure-sdk-for-python
• azure-sdk-for-js

Security Group: "Azure SDK Engineering System Administrators"

Tasks

• Create security group "Azure SDK Engineering System Administrators"
• Add to "Project Collection Administrators" group of "azure-sdk" devops org
• Add to "Owners" group of all 4 azure subs

Design Guidelines: Namespace Factoring

Write design guidelines for namespace factoring in Azure SDKs

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python Guidelines

What are the requirements of the Pipeline?

• Threading
• Can it be expanded to other transports

Add Management plane service and source link table to the readme

Create Azure SDK Design Guidelines

• Architectural Guidelines: Versioning, Compatibility, Packaging, Deployment
• Architectural Guidelines: Code Generation, i.e. relationship to autorest
• Architectural Guidelines: Authorization
• Design Guidelines: Basic API Structure #15
• Design Guidelines: Namespace Factoring #14
• Design Guidelines: Customization of Networking Stack #17
• Design Guidelines: Supporting Cancellations and Timeouts #16
• Design Guidelines: Logging #12
• Design Guidelines: Retries #13
• Design Guidelines: Telemetry #43

Smoke test track 2 packages against live Azure

Requirements

• Create a smoke test in each supported language
• Use the latest public release of the SDK for each supported service
• Use the Mgmt SDK to create resources to test against and to clean up after
• Execute a single request against each supported live service in public Azure

Languages

• .NET
• Java
• Python
• JavaScript

Move schedule triggers from designer into yaml

Our schedule triggers should be moved from the designer into yaml, once they are supported (microsoft/azure-pipelines-yaml#39).

Create starter repo for JavaScript

Add LICENSE

Repos

• azure-sdk-dotnet
• azure-sdk-java
• azure-sdk-python
• azure-sdk-js

Core Concept: eTag

Design Guidelines: Retries

Write design guidelines for supporting retries in Azure SDKs

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python

Use matrix instead of templates for vmImage parallelism

Once matrix variables can be used to specify vmImage (microsoft/azure-pipelines-yaml#4), our yaml files should be converted from job templates to matrix variables.

Ensure we properly disable force push to master for all repos

Disable force push to master even for admins.

• Java repos
• JS repos
• Python Repos
• Net repos

Delete unused DevOps repos

Delete DevOps repos which have been moved to GitHub:

• dotnet
• java
• python

Add CI/Build Badges to repo readmes

• Java
• .NET
• Python
• JavaScript

Disable CI builds of arbitrary branches

Currently, CI builds are run on both pull requests and branches. I think we should keep CI for pull request but disable for arbitrary branches. The latter consumes build agents and adds noise for little benefit. Requiring the intentional act of creating a PR to get CI validation seems like the right balance.

Of course, we may want CI for release branches and similar. This is just disabling CI for dev branches (prior to pull request).

Repos

• dotnet
• java
• python

Ensure email is sent for all build breaks

We need to ensure email is sent for all build breaks. Ideally to the eng sys team, plus commits since last successful build.

Design Guidelines: SDK versioning

Define the versioning strategy for SDK libraries and packages and articulate how service API versions are supported in the SDK.

Different Service Versions, Different client libraries, different Azure clouds (Gov't, Public, etc). How do they map?

Move test.yml to root

Move test.yml from the templates folder to the repo root (as a sibling of azure-pipelines.yml). Having a templates folder is just adding confusion, and the test.yml file will go away once microsoft/azure-pipelines-yaml#4 is fixed.

Repos

• azure-sdk-dotnet
• azure-sdk-java
• azure-sdk-python

Split out language specific Design guidelines for each language

Transient HTTP errors when downloading tools

The downloadTool() function in Azure Pipelines doesn't retry on HTTP errors, so a single transient HTTP error is sufficient to cause a build failure.

Build Logs

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1075

Pipelines Issue

microsoft/azure-pipelines-tool-lib#49

CC: @Azure/azure-sdk-eng

Enable Component Governance Dependency Analyzer

Component Governance is an internal DevOps feature which scans your code to find all dependencies, and issues reports if dependencies have legal or security issues. It should be enabled for all packages in all supported languages. Some of the service SDK teams (e.g. IoT) are already using Component Governance as part of their builds.

Getting Started

https://aka.ms/cgdemovideo
https://aka.ms/cgdocs
https://aka.ms/cgapidocs

Work Items

• Enable CG in all repos
• Ensure team is alerted when CG fails. Failing the build would be best, but may be impossible if CG performs some tasks asynchronously. Next best would be to pass the build but send email when CG fails.
• Understand what types of issues CG does and doesn't detect
• Review existing CG reports for accuract and remaining work

Create DevOps project "public"

• Create DevOps project "public"
• Grant permissions (same as "internal" project for now)
• Add pipelines
  • dotnet
  • java
  • python

Authorization for Azure SDKs needs to be consistent and simple

Update README table with links to source for each service library

Define strategy for running tests in all environments

Define our strategy for running tests in all environments.

Languages & Versions

• dotnet
  • .NET Core 2.2.X
  • .NET Core 2.1.X
  • .NET Framework 4.7.X
• java
  • 7
  • 8
  • 11
• python
  • 2.7
  • 3.5
  • 3.6
  • 3.7
• js
  • node 6
  • node 8
  • node 10
  • node 11
  • browser

Clouds

• prod
• national
• stack
• simulated (localhost)

Test Types

• unit
• functional

Determine how community PRs will be validated

• Infrastructure
  • Azure DevOps (public project)
  • Travis
• Sandboxing
  • How to prevent community PRs from consuming too many resources, accidentally or maliciously
• Community must be able to see results of PR validation

Design Guidelines: Logging

Write design guidelines for supporting logging in Azure SDKs

• .NET Guidelines
• Java Guidelines
• JavaScript Guidelines
• Python

Separate build and test steps in pipeline definitions

Currently, our pipelines do the following:

1. In parallel, build and run tests on all OS and language versions, using the same language version to both build and run tests.
2. If all tests pass, create and publish packages.

Instead, we would prefer the following:

1. Using a single language version (and single OS if possible), build both product and test binaries.
2. If build succeeds, copy product and test binaries to an intermediate location.
3. In parallel, using the pre-built product and test binaries, run tests on all OS and language versions.

A few goals of the new system:

• Publish binaries even if tests fail, since they may be useful for dogfooding, unblocking development, etc.
• Build and run using different language versions. In many languages, this allows using the latest language features in source code while creating packages which can be used from older versions.
• If build step fails, avoid consuming parallel agents for tests which are known to fail.

Repos

• dotnet
• java
• python
• js

Prevent merging pull requests until validation build succeeds

We should prevent merging GitHub pull requests until the Azure Pipelines validation build succeeds.

https://docs.microsoft.com/en-us/azure/devops/pipelines/build/ci-public?view=vsts&tabs=github#protect-branches-with-validation-builds

Repos

• azure-sdk-for-net
• azure-sdk-for-java
• azure-sdk-for-python
• azure-sdk-for-js

Rename DevOps project from "azure-sdk" to "internal"

What's the Testing Story for apps using Azure SDKs?

HttpClient has challenges on Azure App Service

1. Autorest generated code is not doing HttpClient Pooling and exhausting connection limits.
2. (Potential problem) headers are set on HttpClient itself causing reuse issues when pooled.

Create flat folder of all SDK packages and binaries

Requirements

• Collect the latest version of all SDK packages into a flat folder
• Extract the raw binaries (e.g. DLLs for .NET) into a flat folder, for customers who prefer to avoid package managers

Languages

• .NET
• Java
• Python
• JavaScript

Azure SDK documentation needs to be centralized, simple, and high quality

Auto-regenerate sdk based on swagger spec changes

Once changes are made in the corresponding swagger spec at https://github.com/Azure/azure-rest-api-specs , automatic PR with regenerated sdk should be created. Ideally, this should work for all languages.

Define and enforce maximum path length in repo

We should define a maximum path length for files in our repos, then add a check to our CI to enforce the limit.

As an example, the repo https://github.com/Azure/azure-sdk-for-java currently has a file whose path is 233 chars:

azuredatabasemigrationservice/resource-manager/v2018_07_15_preview/
src/main/java/com/microsoft/azure/management/azuredatabasemigrationservice/
v2018_07_15_preview/MigratePostgreSqlAzureDbForPostgreSqlSyncTaskOutputMigrationLevel.java

Windows MAX_PATH is 260 chars, which means you need to clone this repo into a path 27 chars or less.

A reasonable max path for the repo might be say 200 chars, which allows 60 chars for the client prefix.

Enforcement during CI should be fairly easy. One option is a simple shell script to measure the max repo path after cloning and fail if the limit is exceeded. A more thorough option is to intentionally clone and build into a long folder (on Windows). If the clone or build fails, the log should indicate if it was caused by MAX_PATH. This is more thorough because sometimes the files in the repo are under MAX_PATH, but files generated by the build are longer.

Review and update GitHub and DevOps security permissions

Review, update, and document the security permissions applied to our GitHub and DevOps resources.

Resources

• DevOps
  • internal
  • public
    • Building PRs
• GitHub

Rename example from "KeyVault" to "Template"

Repos

• azure-sdk-dotnet
• azure-sdk-java
• azure-sdk-python