azure / apiops

APIOps applies the concepts of GitOps and DevOps to API deployment. By using practices from these two methodologies, APIOps can enable everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they’re building.

Code Of Conduct

Before we dive into learning about the APIOPs tool we would like to share some guidance on how to navigate the associated Github repo as you learn your way around the tool:

• Always refer to the wiki for detailed configuration setup
• Always use the latest release which includes the latest features and bug fixes
• Please use the issues section to report any issues that you encounter as well as any requested features that you would like to raise to the development team. Always remember to close your issues when its gets resolved
• Remember to subscribe to this repo (at least subscribe to be notified about new releases) to stay in the loop as we are always adding new features and squashing these pesky bugs
• Please read this guide before opening any issues
• This is an open source project and hence we are always accepting contributions from the community so make sure you fork and submit your PRs. We take them seriously and they help us make the tool better

Video Guide

Please note that the videos below serve as a good starting point but it is recommended to spend some time reading the detailed documentation found under the Github page and the wiki.

Check this video for a 360 overview of the APIOPS tool.

Check this video for a step by step guide that demonstrates setting up a Dev,QA, and Prod envirionments in Azure DevOps. Keep in mind that we also support Github even though its not demonstrated in this video.

Versions

Please note that starting with release V.3.0.0 we changed the way the updates to the system get delivered. Thus when reading the docs you will notice that there are some sections that are designated with the <= V.3.0.0 and >= V.3.0.0 suffixes to distinguish between the old and the new update delivery methodology. We recommend switching to the new methodology whenever possible as it greatly simplifies fetching the latest version of the code.

Old method (version < 3.0.0)

Each time a new release was pushed you would have to:

• Download the updated code folder as well as the updated pipelines
• You would have to run the publish-extractor and publish-publisher pipelines and store the resulting binaries in your own internal feed
• We dropped the publish-publisher and publish-extractor pipelines as we now host the binaries for you. If you would rather host the binaries yourself you still have access to the source code which you can download and build yourself. Just keep in mind that these deprecated pipelines now live in the releases themselves and won't be found in the repository.

New method (version >= 3.0.0)

• The binaries are now hosted for you on the public github repo as part of the release
• Utilizing the new version simply involves pointing the pipelines to the new version by changing environment variables (refer to the docs on how to achieve that). No more rebuilding or hosting of binaries
• You only have to download the updated pipelines if they are updated. As part of the release we now push two different sets of pipelines. The Azure_DevOps.zip if you are a Azure Devops user and Github.zip if you are a Github user
• If you would still rather host the extractor and publisher binaries within your own environment you have two options:
  • We publish the compiled publisher and extractor binaries as part of the release so you can download these and host them internally if you want
  • The publish-publisher and publish-extractor pipelines are now legacy pipelines that can be found under the V.2.x release if you still need access to them. Also note that we still include the source code as part of the Source code (zip) in each release

About this Tool

APIOps applies the concepts of DevOps to Azure API Management. This enables everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they're building. APIOps places the Azure API Management infrastructure under version control to achieve these goals. Rather than making changes directly in API Management portal, most operations happen through code changes that can be reviewed and audited. In this section, we include links to both a complementary Guide and Wiki to get you started with the tool.

Please bear in mind that APIOPS is designed to facilitate the promotion of changes across different Azure API Management (APIM) instances. While the animation below illustrates changes within the same instance, it's important to note that you can effortlessly apply your modifications across various Azure APIM instances using the supported configuration system. We advise taking some time to explore the wiki and documentation to grasp the functioning of configuration overrides when promoting changes across different environments.

Complementary Guide

This guide is designed to bring a 400-level understanding of automating API deployments in Azure Api Management. This is meant to be a hands-on lab experience, all instructions are provided, but a basic level of understanding of apis, devops and gitops is expected. This guide will guide you through the concept of APIOps. It applies the concepts of GitOps and DevOps to API deployment. By using practices from these two methodologies, APIOps can enable everyone involved in the lifecycle of API design, development, and deployment with self-service and automated tools to ensure the quality of the specifications and APIs that they're building. This Guide is delivered using Github Pages and Just-The-Docs theme.

Wiki

The Wiki covers all of our supported resources. Think about it as being more resource-focused with deeper dives, as opposed to the aforementioned guide which is more scenario-based and holistic. So in essence you need to read both to have a successful adoption of the tool. The wiki can be found on wiki tab within this repo or by following this link.

Roadmap

To stay up-to-date with the APIOps roadmap, please follow it closely. To stay updated on the APIOps roadmap, please make sure to monitor it closely here.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Supporting Tools

While the APIOPs tool does not have built-in support for promoting the migration of the APIM Dev Portal, there is another tool available that offers such functionality. We suggest you explore that tool which can be found here.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

apiops's Issues

Modify the order of operations when it comes to gateways and products.

Basically the code will be modified to make sure that the apis exists before applying the aforementioned entities

Azure Devops pipeline succeeding in spite of exit code 255

creator tool exits with code 255 but the bash exits with code 0 which leads to a false positive. Need to further investigate.

Add the new APIM feature for Policy Fragments

Hello there,

I'd like to request adding the new APIM feature, Policy Fragments, as a supported feature for this tool. I tried running it with a policy fragment.

Policy Fragment (named ForwardContext):
<fragment> <set-header name="x-request-context-data" exists-action="override"> <value>@(context.Deployment.Region)</value> </set-header> </fragment>

API Policy using this fragment:
<inbound> <include-fragment fragment-id="ForwardContext" /> <base /> </inbound>

As expected, the extractor pulled in the policy, but not the fragment. The publisher failed as it was expecting the fragment to be part of APIM and it wasn't.

Publisher Error:
{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"include-fragment","message":"Error in element 'include-fragment' on line 15, column 4: Policy fragment with id 'ForwardContext' could not be found."}]}}

Thanks and please let me know if you need further details.

serviceinformation override on code promotion

The serviceinformation file from the source environment is carries over to all environments during code promotion thereby causing the publisher to fail when the target apim location is different.

Moreover each environment will have its own configuration across environments and needs to have a provision for all service configuration to be overridden.

Modify the order of operations when it comes to applying application insights on the api level

Creator tries to apply the application insights settings on the api level before creating the api in the target environment which causes the deployment to fail.

Need to restore the git logic that detects changes in the creator tool

• Changes via git. Right now it pushes everything in the folder. Doesn't look at the commit id and pushes what was changed.
• If you delete something from the source it doesn't get deleted becuase of the git change functionality that is still missing.

How do we Handle Secrets from Environment to Another Environment

Hi,
When I used the Extract and Pulled the Named values, for the secret it extracts like below

{
"name": "Sec",
"properties": {
"displayName": "Sec",
"secret": true
}
}
When I Publish it I get the Error as it doesn't have any value.
How can I Publish these kind of values?

How to handle key vault integration in named parameters for multiple environments

Hi there,

I'm not seeing a way to allow for handling key vault integration in the named parameters in the configuration..yaml between environments. For example, it looks like the only properties available to set on named parameters in the config file are:

name:
displayName:
value:

If you have your named value tied to a key vault there are a few different properties that need to be available:

keyVault (object): {secretIdentifier:}
secret (bool?):

Is this something that could be added to your backlog?

Thanks!

Run the Publisher with the Commit ID

HI,

I want to run the Run-Publisher Pipleine with the specific Commit _ID.

i saw in the documentation we can pass it in the COMMIT_ID Paramenter. so i tried to do it in the Pipleine by specifying in the Parameters like below

parameters:
COMMIT_ID: "a2a7ab5e790affcd8f6abfd8a94ba931e49c580a"
API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH: ${{parameters.API_MANAGEMENT_SERVICE_OUTPUT_FOLDER_PATH }}
RESOURCE_GROUP_NAME : $(RESOURCE_GROUP_NAME_Prod)
CONFIGURATION_YAML_PATH: $(Build.SourcesDirectory)/configuration.prod.yaml
ENVIRONMRNT: "Prod"

But i get the below Error

/tools/pipelines/run-publisher.yaml (Line: 50, Col: 33): Unexpected parameter 'COMMIT_ID'

Can you Please help me with it?

Add support for overriding an api serviceurl

Add support for overriding an api serviceurl when promoting across environments.

Backend id not extracted resulting in error on publish

created a new api using azure function in the backend. it has a function key
when running the publisher i get an error like this
System.InvalidOperationException: REST API call failed. Status code is 400, response content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"set-backend-service","message":"Error in element 'set-backend-service' on line 4, column 4: Backend with id 'ahmsdemofunc' could not be found."}]}}'.

Need more docs/clarity on what needs to be done so I don't get such an error?

Trigger publisher on configuration file change

Currently the publisher only publishes changes to the apim instance when it detects changes in the artifacts folder. Need to modify the behavior to also trigger when the configuration file is modified.

Versioned APIs generating apiversionset.json even when not being included in the list of apis to be extracted

When extracting select apis using the extractor configuration file, if there is a versioned apis that is not included in the list of apis to be extracted then it always end up with a file called apiversionset.json in the repository. This comes into play under scenarios where we are supporting independent api teams.

This will create confusion as the api has no business showing up in the repository if its not included in the extractor yaml configuration file

Adding support for Swagger (OpenApi 2.0)

Currently apiops only supports OpenAPI 3 JSON & YAML files. APIM on the other hand supports inline swagger (OpenAPI 2.0), WADL-XML, and WSDL documents.

We can add support for it . The tricky part is there are some scenarios where our tool needs to parse the specification file to get data. The amount of effort required for support will depend on how easily we can parse that file format.

Handle conflict between apiInformation.json and specification.yaml

Certain API properties are exposed in both apiInformation.json and specification.yaml (displayName, description, etc). The current implementation prioritizes the specification.yaml. This means that users can modify the JSON file but have their changes overwritten by the YAML file. Here are a few potential solutions:

• Do a commit pre-check and reject the commit if apiInformation.json and specification.yaml have conflicting information. Would only work for GitHub; don't think ADO supports this.
• Modify the format of apiInformation.json so that duplicate properties are only contained in specification.yaml. This would ensure one source of truth. It would require users to maintain a specification.yaml though, as some of these duplicate properties are mandatory. This might not be a bad thing; start with an OpenAPI spec, and only use apiInformation.json for APIM-specific info.

Application insights credentials error when communicating with the apim restful api

Working on resolving application insights credentials error when communicating with the apim restful api.

Modify create_pull_request.sh to support config files in Azure DevOps

Currently the Azure DevOps pipeline fails when there is a configuration lives in the root of the repository. This needs to be modified to allow config.yaml file to coexsit.

Renaming an api on the portal and then extracting it causes the api to get deleted upon publishing

Steps to recreate:

Extract -> Rename -> Extract -> Publish

1- Extract
1- rename an api on the portal
2- run the extractor
3- run the publisher

api gets deleted.

Build Azure Devops and Github Pipelines to promote code across environemnts

Try an end to end promotion pipeline to see if we can successfully move between environments (e.g. dev -> QA-> prod) where each environment utilizes a different APIM instance.

serviceinformation overrides custom domains on apipromotion

When after pulling apiops latest code and working with apiops pipeline by running full publisher the service information overrides the custom domains provisioned through terraform still the issue persists.Please figure out why this behavior occuring.

Exit code 0 on failure

System.InvalidOperationException: REST API call failed. Status code is 409, response content is '{"error":{"code":"ServiceAlreadyExists","message":"Api service already exists: apim-prod-wk-03022022","details":null,"innerError":null}}'.
at common.AzureHttpExtensions.ValidateSuccess(Response response) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 147
at common.AzureHttpClient.PutJsonObject(Uri uri, JsonObject jsonObject, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 75
at creator.Creator.PutService(Service service, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 204
at creator.Creator.PutServiceInformationFile(ServiceInformationFile file, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 198
at creator.Creator.PutServiceInformationFile(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 180 at creator.Creator.PutFiles(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 164
at creator.Creator.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 102
at creator.Creator.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 72
info: Microsoft.Hosting.Lifetime[0]
Application is shutting down...
fail: Microsoft.Extensions.Hosting.Internal.Host[9]
BackgroundService failed
System.InvalidOperationException: REST API call failed. Status code is 409, response content is '{"error":{"code":"ServiceAlreadyExists","message":"Api service already exists: apim-prod-wk-03022022","details":null,"innerError":null}}'.
at common.AzureHttpExtensions.ValidateSuccess(Response response) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 147
at common.AzureHttpClient.PutJsonObject(Uri uri, JsonObject jsonObject, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 75
at creator.Creator.PutService(Service service, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 204
at creator.Creator.PutServiceInformationFile(ServiceInformationFile file, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 198
at creator.Creator.PutServiceInformationFile(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 180 at creator.Creator.PutFiles(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 164
at creator.Creator.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 102
at creator.Creator.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 72
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
crit: Microsoft.Extensions.Hosting.Internal.Host[10]
The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
System.InvalidOperationException: REST API call failed. Status code is 409, response content is '{"error":{"code":"ServiceAlreadyExists","message":"Api service already exists: apim-prod-wk-03022022","details":null,"innerError":null}}'.
at common.AzureHttpExtensions.ValidateSuccess(Response response) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 147
at common.AzureHttpClient.PutJsonObject(Uri uri, JsonObject jsonObject, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 75
at creator.Creator.PutService(Service service, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 204
at creator.Creator.PutServiceInformationFile(ServiceInformationFile file, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 198
at creator.Creator.PutServiceInformationFile(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 180 at creator.Creator.PutFiles(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 164
at creator.Creator.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 102
at creator.Creator.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 72
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
Exit code is 0

Publisher failing to create namevalue property in the correct order

If the namevalue property is used within a policy which is applied at the global scope then a http 400 error is being thrown. This doesn't happen when creating a namevalue property that is used within a policy at the api scope. Seems like the restful api calls may be executing in the incorrect order where its trying to apply the global policy before creating the namevalue property.

api service Information with custom domain name is not retrieved

When working with new api custom domains for apim instance,the extractor is not fetching and updating the information with workflow when it runs the extractor it only takes already configured proxy's but not the new custom domains.

Need to modify the extractor tool to recognize name/values

Tool doesn't recognize the name/values which makes it challenging to override the values across environments.

API overrides needs to specify diagnostic overrides for every version - revision pair

I will update with more information later. Here's a quick summary.

If you have a versioned API that has diagnostic loggers, the Configuration file needs to override the diagnostic logger resource id for every version-revision pair or the deployment will fail.

Steps to reproduce:

• Add a new revision or version to an existing API.
• Configure some diagnostic logger for that new version/revision.
• Run extractor, approve PR, deploy to next environment
• The deployment will fail, since the Configuration doesn't override the diagnostic logger for the specific version/revision (unless the same logger exists in both APIM instances).

Workaround:
If adding new revisions or versions with diagnostic loggers, update the generated PR to add an API override section for the new version/revision before merging.

Publisher fails to create product and api on promotion

Steps to reproduce:

1. Create new Product_v1 in ENV_SBX
2. Create new API_v1 in ENV_SBX
3. Link API_v1 -> Product_v1 in ENV_SBX
4. Promote/Copy files from Product_v1 and API_v1 to new environment/instance ENV_NONPROD
5. When publisher runs in ENV_NONPROD, it detects changes in git commit and tries to apply those changes.
6. Order in which it tries to apply changes: (create new product -> create new API)
7. Fails because product has linking to API which has not been created at this point.
8. Product_v1 gets created in ENV_NONPROD but without any APIs.

Error Log:

info: creator.Creator[0]
      Beginning execution...
info: creator.Creator[0]
      Updating Azure service policy with file /<GIT-REPO-NAME-HERE>/APIM-NONPROD/policy.xml...
info: creator.Creator[0]
      Updating Azure service product with file /<GIT-REPO-NAME-HERE>/APIM NONPROD/products/Unlimited/productInformation.json...
info: creator.Creator[0]
      Updating Azure service product with file /<GIT-REPO-NAME-HERE>/APIM-NONPROD/products/demo-apim-product-aman/productInformation.json...
info: creator.Creator[0]
      Updating Azure service product with file /<GIT-REPO-NAME-HERE>/APIM-NONPROD/products/Starter/productInformation.json...
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /<GIT-REPO-NAME-HERE>/
crit: creator.Creator[0]
      REST API call failed. Status code is 400, response content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"aid","message":"API not found"}]}}'.
      System.InvalidOperationException: REST API call failed. Status code is 400, response content is '{"error":{"code":"ValidationError","message":"One or more fields contain incorrect values:","details":[{"code":"ValidationError","target":"aid","message":"API not found"}]}}'.
         at common.ArmClientExtensions.ValidateSuccess(Response response)
         at common.ArmClientExtensions.<>c.<PutResource>b__5_3(Response response)
         at common.TaskExtensions.Map[T,U](Task`1 task, Func`2 mapper)
         at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int[32]() permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
         at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
         at common.ArmClientExtensions.PutResource(ArmClient client, Uri uri, Stream contentStream, CancellationToken cancellationToken)
         at creator.Creator.PutProductApi(ProductUri productUri, ApiName apiName, CancellationToken cancellationToken)
         at common.TaskExtensions.Bind[T,U](Task`1 task, Func`2 binder)
         at creator.Creator.SetProductApis(FileInfo file, CancellationToken cancellationToken)
         at creator.Creator.PutProductInformation(FileInfo file, CancellationToken cancellationToken)
         at common.EnumerableExtensions.<>c__DisplayClass4_0`1.<<ExecuteInParallel>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at common.EnumerableExtensions.ExecuteInParallel[T](IEnumerable`1 source, Func`3 action, CancellationToken cancellationToken)
         at creator.Creator.ProcessFilesToPut(ILookup`2 fileLookup, CancellationToken cancellationToken)
         at creator.Creator.ProcessFiles(ImmutableDictionary`2 fileMap, CancellationToken cancellationToken)
         at common.TaskExtensions.Bind[T,U](Task`1 task, Func`2 binder)
         at creator.Creator.ExecuteAsync(CancellationToken cancellationToken)
         at common.ConsoleService.<StartAsync>b__6_0()
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...

CC: @seenu433, @sinamas1, @filidav, @pranaysharmamanulife

Add support to the extractor tool to support multiple teams working on different apis

Option to process serviceinformation.json

In my testing, ServiceInformation.json is always processed and results in my actual APIM instance being updated. I am not sure why this is the case. This takes more than 30 mins - and will be a big no no for many customers and especially prod environment. It also means the pipeline is sitting for 30 mins plus

Extractor fails when creating folder names with invalid characters

Add an operation with the below

Display Name: /Box/{type}/{Id}/content - GET
URL GET /Box/{type}/{Id}/content

Add an inbound policy at the operation

Run extractor

Introduce ability to provide custom configuration across different deployment environments

We need to modify the creator tool to allow parsing a custom configuration file to allow deployment across different environments while allowing custom configuration for the following items:

• Policies (xml files)
• apiinformation.json
• specification.yaml
• etc.

We will have to utilize name/value pairs as well as the context object within policies.
For other configurations like service urls we will have to replace the value in the files.

Add Integration Testing

As we start to implement new features and have more developers, unit tests will be necessary to avoid breaking functionality.
Also, this tool is provided as-is, without Microsoft support (outside of the team here). It will encourage developer adoption.

Extractor tool injecting new line feed into the policy.xml file when used with multiline policies (e.g. cache-store policy)

is line feed. It may need to be escaped. Creator tool fails with an error code 400 when it tries to parse the file.

Missing support for Access Control of Products

Unable to extract / publish Access control for Products for Groups / AD Groups.

Steps to reproduce:

Map a product to an Azure AD Group
Run Extractor
productInformation.json is missing the access control for that group

Add support to update the developer portal between environments

The original extractor tool had the benefit of having all the developer portal templates, which made it easy to update the developer portal between environments

Investigating the ability to get the developer portal info with the standard REST API calls to APIM.

option to run with/without commit id

For testing and some scenarios, allow the commit id check to be configurable

Support Version Sets

Looks like this may not support version sets at the moment. I can provide more details if needed. I may be able to add the support if it's not already in progress and not out of scope.
Reference repo: https://github.com/colincmac/oink-azure-apim-publisher-apiops

Error:

48 BackgroundService failed 

49 System.InvalidOperationException: REST API call failed. Status code is 404, response content is '{"error":{"code":"ResourceNotFound","message":"ApiVersionSet not found.","details":null}}'. 

Race condition when dealing with API Revisions

When exporting apis with multiple revisions, you can get a race condition trying to output the apis to disk.

I am able to recreate this using an apim instance with 1 api and multiple revisions.

As part of the publish-extractor and publish-publisher pipelines should we add release numbers in the binaries

Add the Assembly task in our pipelines. That way our binaries will be versioned and then if the development team makes a change and does a new build they will get their own build number

Api level diagnostics settings failing to apply successfully on higher environemnt

Upon further investigation we realized that the settings is being applied after returning an http status code 202 (request accepted) instead of waiting for http status code 201 (created)

fail: Microsoft.Extensions.Hosting.Internal.Host[9]
BackgroundService failed
System.InvalidOperationException: REST API call failed. Status code is 404, response content is '{"error":{"code":"ResourceNotFound","message":"Api not found.","details":null}}'.
at common.AzureHttpExtensions.ValidateSuccess(Response response) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 154
at common.AzureHttpClient.PutJsonObject(Uri uri, JsonObject jsonObject, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/common/AzureHttpClient.cs:line 75
at common.ApiDiagnostic.Put(Func4 putResource, ServiceProviderUri serviceProviderUri, ServiceName serviceName, ApiName apiName, ApiDiagnostic apiDiagnostic, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/common/ApiDiagnostic.cs:line 152 at creator.Creator.PutApiDiagnosticInformationFile(ApiDiagnosticInformationFile file, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 910 at System.Threading.Tasks.Parallel.<>c__501.<b__50_0>d.MoveNext()
--- End of stack trace from previous location ---
at creator.Creator.PutApiDiagnosticInformationFiles(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 876 at creator.Creator.PutFiles(IReadOnlyCollection1 files, CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 251
at creator.Creator.Run(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 113
at creator.Creator.ExecuteAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/tools/code/creator/Creator.cs:line 79
at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

serviceinformation with custom domains overwritem

When working with custom domains, the extractor don't get that information, therefore when the publish runs it get rid of all custom domains you have configured

maybe related to the issue #46

Updating both apiInformation.json and specification.yaml being required when using Publisher is not obvious

Updates to an API may only require updating apiInformation.json or specficiation.yaml. Publisher expects both to be updated in tandem due to the grouping/joining within the method PutApiInformationAndSpecificationFiles in Publisher.cs.

Publisher fails when the source and target environment have different regions

Publisher picks up the dev environment settings in serviceInformation.json and tries to apply this to the prod environment. If the prod environment was created in a different regions then it will generate an error.

Unable to deploy the changes inside specification.yaml file

Hello,

We tried following the documentation and can able to extract the APIs to our Repo.
However, only the changes inside apiinformation.json is getting reflected in APIM and not the specification.yaml.

Updating configuration.<env>.yaml file ONLY doesn't publish changes to expected environment

Hello all,

During my testing I noticed that if I only change the configuration.prod.yaml file and commit that change it doesn't update the environment as I would expect. The reason seems to be related to the fact that nothing actually changed in the artifacts/named values/ folders.

As a workaround, I was able to update one of the namedValueInformation.json files along with the configuration.prod.yaml file in a single commit and that change DID get applied to the prod environment as expected. This is not a workable solution in my opinion as it will also update the lower environment since the namedValueInformation.json file had to be updated.

Doesn't work when this is included in the commit:
apiops/configuration.prod.yaml -> file changed and committed but nothing published to the prod environment.

Works when this is included in the commit:
apiops/artifacts/named values/environment/namedValueInformation.json -> file changed, committed, and published to lower environment
apiops/configuration.prod.yaml -> file changed, committed, and published to higher environment

Let me know if you need additional details.

Update the documentation

Update the documentation to reflect the feedback from the workshop

Update the documentation to reflect the changes introduced to allow custom configurations across different environments

Modify the creator tool to pick changes from overlay folder

We need to modify the extractor tool to detect commit changes from the overlay folder as currently the tool only detects changes in the artifacts folder. This will be required to detect both changes in the custom policy files as well as the configuration file.

Creator doesn't deploy the operations via code

The creator doesn't deploy the operations although they are included specification.yaml. This means that the creator never really worked as intended as this means if someone deleted the operation by mistake from the portal they can't recover it via code. This also explains why when promoted to a higher environment the operations never get published.

Revisit ArmClient implementation

We currently use HttpPipelines in the ArmClient class to make Azure REST API calls. These pipelines handle many things automatically (authentication, retries, etc). However, as of this change, the Azure.ResourceManager.Resources library no longer exposes HttpPipelines for external use. This leaves us with two options:

• Do nothing, which means we're stuck with Azure.ResourceManager.Resources version 1.0.0-beta.3.
• Use another solution for making Azure REST API calls.

Need to update docs to include AzDO pipeline permissions for repositories

AzDO Build pipeline agents don't have permission by default to contribute to a repo, create a branch or update a pr.
You need to grant that permission, This might be something good to add to docs.

Run-extractor pipeline was failing with (A potentially dangerous Request.Path value was detected from the client) while creating the PR

Hi Team,

While running the extractor pipeline in azure devops , Pipelines are getting failed in creating PR task. Please find the below logs regarding to issue and please help me out to fix the issue.

Error log:

Creating folder /home/vsts/work/_temp/artifacts-from-portal...
Cloning branch main...
ERROR: A potentially dangerous Request.Path value was detected from the client (:).
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[debug]task result: Failed
##[error]Script failed with exit code: 1

Complete log:

2022-05-30T05:03:52.2780624Z ##[debug]Evaluating condition for step: 'Create pull request with artifacts'
2022-05-30T05:03:52.2783177Z ##[debug]Evaluating: SucceededNode()
2022-05-30T05:03:52.2783688Z ##[debug]Evaluating SucceededNode:
2022-05-30T05:03:52.2784975Z ##[debug]=> True
2022-05-30T05:03:52.2785654Z ##[debug]Result: True
2022-05-30T05:03:52.2786314Z ##[section]Starting: Create pull request with artifacts
2022-05-30T05:03:52.2802285Z ==============================================================================
2022-05-30T05:03:52.2802626Z Task : Azure CLI
2022-05-30T05:03:52.2803111Z Description : Run Azure CLI commands against an Azure subscription in a PowerShell Core/Shell script when running on Linux agent or PowerShell/PowerShell Core/Batch script when running on Windows agent.
2022-05-30T05:03:52.2803645Z Version : 2.198.0
2022-05-30T05:03:52.2803859Z Author : Microsoft Corporation
2022-05-30T05:03:52.2804174Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-cli
2022-05-30T05:03:52.2804597Z ==============================================================================
2022-05-30T05:03:52.3529832Z ##[debug]Using node path: /home/vsts/agents/2.204.0/externals/node10/bin/node
2022-05-30T05:03:52.5890146Z ##[debug]agent.TempDirectory=/home/vsts/work/_temp
2022-05-30T05:03:52.5907830Z ##[debug]loading inputs and endpoints
2022-05-30T05:03:52.5919350Z ##[debug]loading INPUT_CONNECTEDSERVICENAMEARM
2022-05-30T05:03:52.5934014Z ##[debug]loading INPUT_SCRIPTTYPE
2022-05-30T05:03:52.5935986Z ##[debug]loading INPUT_SCRIPTLOCATION
2022-05-30T05:03:52.5937000Z ##[debug]loading INPUT_SCRIPTPATH
2022-05-30T05:03:52.5937912Z ##[debug]loading INPUT_SCRIPTARGUMENTS
2022-05-30T05:03:52.5938816Z ##[debug]loading INPUT_POWERSHELLERRORACTIONPREFERENCE
2022-05-30T05:03:52.5939743Z ##[debug]loading INPUT_ADDSPNTOENVIRONMENT
2022-05-30T05:03:52.5940621Z ##[debug]loading INPUT_USEGLOBALCONFIG
2022-05-30T05:03:52.5941464Z ##[debug]loading INPUT_CWD
2022-05-30T05:03:52.5942303Z ##[debug]loading INPUT_FAILONSTANDARDERROR
2022-05-30T05:03:52.5943219Z ##[debug]loading INPUT_POWERSHELLIGNORELASTEXITCODE
2022-05-30T05:03:52.5944841Z ##[debug]loading ENDPOINT_AUTH_d3bd7077-d12c-4178-b66f-2985165c7989
2022-05-30T05:03:52.5946022Z ##[debug]loading ENDPOINT_AUTH_SCHEME_d3bd7077-d12c-4178-b66f-2985165c7989
2022-05-30T05:03:52.5947200Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_d3bd7077-d12c-4178-b66f-2985165c7989_TENANTID
2022-05-30T05:03:52.5948404Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_d3bd7077-d12c-4178-b66f-2985165c7989_SERVICEPRINCIPALID
2022-05-30T05:03:52.5949693Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_d3bd7077-d12c-4178-b66f-2985165c7989_AUTHENTICATIONTYPE
2022-05-30T05:03:52.5951287Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_d3bd7077-d12c-4178-b66f-2985165c7989_SCOPE
2022-05-30T05:03:52.5952492Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_d3bd7077-d12c-4178-b66f-2985165c7989_SERVICEPRINCIPALKEY
2022-05-30T05:03:52.5953524Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2022-05-30T05:03:52.5954445Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2022-05-30T05:03:52.5955444Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2022-05-30T05:03:52.5956382Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2022-05-30T05:03:52.5957213Z ##[debug]loaded 22
2022-05-30T05:03:52.5958478Z ##[debug]Agent.ProxyUrl=undefined
2022-05-30T05:03:52.5959330Z ##[debug]Agent.CAInfo=undefined
2022-05-30T05:03:52.5960179Z ##[debug]Agent.ClientCert=undefined
2022-05-30T05:03:52.5961069Z ##[debug]Agent.SkipCertValidation=undefined
2022-05-30T05:03:52.5972461Z ##[debug]check path : /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.198.0/task.json
2022-05-30T05:03:52.5975292Z ##[debug]adding resource file: /home/vsts/work/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/2.198.0/task.json
2022-05-30T05:03:52.5976579Z ##[debug]system.culture=en-US
2022-05-30T05:03:52.5987431Z ##[debug]which 'az'
2022-05-30T05:03:52.5996064Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:03:52.6008042Z ##[debug]scriptType=bash
2022-05-30T05:03:52.6009175Z ##[debug]scriptLocation=scriptPath
2022-05-30T05:03:52.6012424Z ##[debug]scriptArguments=--organization-url "https://dev.azure.com/naganu836/" --project-name "williams-poc" --repository-name "https://[email protected]/naganu836/williams-poc/_git/apiops.git" --pull-request-title "Merging artifacts from portal (Build 180)" --branch-name "main" --source-folder-path "/home/vsts/work/1/artifacts-from-portal" --temporary-branch-name "artifacts-from-portal-build-180" --temporary-folder-path "/home/vsts/work/_temp/artifacts-from-portal" --overwrite-subfolder "extract"
2022-05-30T05:03:52.6016750Z ##[debug]scriptPath=/home/vsts/work/1/s/tools/utils/create_pull_request.sh
2022-05-30T05:03:52.6025219Z ##[debug]which 'bash'
2022-05-30T05:03:52.6027958Z ##[debug]found: '/usr/bin/bash'
2022-05-30T05:03:52.6030491Z ##[debug]which '/usr/bin/bash'
2022-05-30T05:03:52.6031166Z ##[debug]found: '/usr/bin/bash'
2022-05-30T05:03:52.6032905Z ##[debug]/usr/bin/bash arg: /home/vsts/work/1/s/tools/utils/create_pull_request.sh
2022-05-30T05:03:52.6034531Z ##[debug]/usr/bin/bash arg: --organization-url "https://dev.azure.com/naganu836/" --project-name "williams-poc" --repository-name "https://[email protected]/naganu836/williams-poc/_git/apiops.git" --pull-request-title "Merging artifacts from portal (Build 180)" --branch-name "main" --source-folder-path "/home/vsts/work/1/artifacts-from-portal" --temporary-branch-name "artifacts-from-portal-build-180" --temporary-folder-path "/home/vsts/work/_temp/artifacts-from-portal" --overwrite-subfolder "extract"
2022-05-30T05:03:52.6039629Z ##[debug]cwd=/home/vsts/work/1/s
2022-05-30T05:03:52.6040235Z ##[debug]scriptLocation=scriptPath
2022-05-30T05:03:52.6040845Z ##[debug]failOnStandardError=false
2022-05-30T05:03:52.6041585Z ##[debug]testing directory '/home/vsts/work/1/s'
2022-05-30T05:03:52.6072274Z ##[debug]which 'az'
2022-05-30T05:03:52.6074575Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:03:52.6076560Z ##[debug]/usr/bin/az arg: --version
2022-05-30T05:03:52.6077282Z ##[debug]/usr/bin/az arg: --version
2022-05-30T05:03:52.6079702Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:52.6080553Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:52.6081119Z ##[debug]arguments:
2022-05-30T05:03:52.6081644Z ##[debug]arguments:
2022-05-30T05:03:52.6082276Z ##[debug] --version
2022-05-30T05:03:52.6082906Z ##[debug] --version
2022-05-30T05:03:52.6083862Z [command]/usr/bin/az --version
2022-05-30T05:03:54.5017317Z WARNING: You have 2 updates available. Consider updating your CLI installation with 'az upgrade'
2022-05-30T05:03:54.5018007Z azure-cli 2.36.0 *
2022-05-30T05:03:54.5019251Z
2022-05-30T05:03:54.5019370Z
2022-05-30T05:03:54.5019915Z Please let us know how we are doing: https://aka.ms/azureclihats
2022-05-30T05:03:54.5021669Z and let us know if you're interested in trying out our newest features: https://aka.ms/CLIUXstudy
2022-05-30T05:03:54.5022098Z core 2.36.0 *
2022-05-30T05:03:54.5022706Z telemetry 1.0.6
2022-05-30T05:03:54.5022970Z
2022-05-30T05:03:54.5023350Z Extensions:
2022-05-30T05:03:54.5023924Z azure-devops 0.25.0
2022-05-30T05:03:54.5024205Z
2022-05-30T05:03:54.5024587Z Dependencies:
2022-05-30T05:03:54.5024968Z msal 1.17.0
2022-05-30T05:03:54.5025574Z azure-mgmt-resource 20.0.0
2022-05-30T05:03:54.5025878Z
2022-05-30T05:03:54.5026413Z Python location '/opt/az/bin/python3'
2022-05-30T05:03:54.5027067Z Extensions directory '/opt/az/azcliextensions'
2022-05-30T05:03:54.5027406Z
2022-05-30T05:03:54.5027867Z Python (Linux) 3.8.13 (default, Apr 22 2022, 04:53:09)
2022-05-30T05:03:54.5057721Z [GCC 9.4.0]
2022-05-30T05:03:54.5058405Z
2022-05-30T05:03:54.5058876Z Legal docs and information: aka.ms/AzureCliLegal
2022-05-30T05:03:54.5059188Z
2022-05-30T05:03:54.5059434Z
2022-05-30T05:03:54.5061121Z ##[debug]useGlobalConfig=false
2022-05-30T05:03:54.5062175Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp
2022-05-30T05:03:54.5063832Z ##[debug]Agent.TempDirectory=/home/vsts/work/_temp
2022-05-30T05:03:54.5084418Z Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask
2022-05-30T05:03:54.5087002Z ##[debug]connectedServiceNameARM=d3bd7077-d12c-4178-b66f-2985165c7989
2022-05-30T05:03:54.5088475Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 data environment = AzureCloud
2022-05-30T05:03:54.5089119Z Setting active cloud to: AzureCloud
2022-05-30T05:03:54.5090017Z ##[debug]which 'az'
2022-05-30T05:03:54.5121232Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:03:54.5166631Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud
2022-05-30T05:03:54.5167780Z ##[debug]/usr/bin/az arg: cloud set -n AzureCloud
2022-05-30T05:03:54.5168589Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:54.5169330Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:54.5170094Z ##[debug]arguments:
2022-05-30T05:03:54.5170781Z ##[debug]arguments:
2022-05-30T05:03:54.5171901Z ##[debug] cloud
2022-05-30T05:03:54.5172634Z ##[debug] cloud
2022-05-30T05:03:54.5173322Z ##[debug] set
2022-05-30T05:03:54.5174032Z ##[debug] set
2022-05-30T05:03:54.5175223Z ##[debug] -n
2022-05-30T05:03:54.5175988Z ##[debug] -n
2022-05-30T05:03:54.5176539Z ##[debug] AzureCloud
2022-05-30T05:03:54.5177100Z ##[debug] AzureCloud
2022-05-30T05:03:54.5177660Z [command]/usr/bin/az cloud set -n AzureCloud
2022-05-30T05:03:56.5001147Z ##[debug]connectedServiceNameARM=d3bd7077-d12c-4178-b66f-2985165c7989
2022-05-30T05:03:56.5015964Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 auth scheme = ServicePrincipal
2022-05-30T05:03:56.5017343Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 data SubscriptionID = 195ae8df-2d72-46f3-8d91-e83a1401efd2
2022-05-30T05:03:56.5018613Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 auth param authenticationType = spnKey
2022-05-30T05:03:56.5020395Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 auth param serviceprincipalid = ***
2022-05-30T05:03:56.5021645Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 auth param tenantid = 9ad0e722-d594-42e3-8fd3-34389ad0b720
2022-05-30T05:03:56.5023402Z ##[debug]key based endpoint
2022-05-30T05:03:56.5024952Z ##[debug]d3bd7077-d12c-4178-b66f-2985165c7989 auth param serviceprincipalkey = ***
2022-05-30T05:03:56.5030768Z ##[debug]Processed: ##vso[task.setsecret]***
2022-05-30T05:03:56.5031830Z ##[debug]which 'az'
2022-05-30T05:03:56.5032564Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:03:56.5033869Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "9ad0e722-d594-42e3-8fd3-34389ad0b720" --allow-no-subscriptions
2022-05-30T05:03:56.5035375Z ##[debug]/usr/bin/az arg: login --service-principal -u "" --password= --tenant "9ad0e722-d594-42e3-8fd3-34389ad0b720" --allow-no-subscriptions
2022-05-30T05:03:56.5036520Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:56.5037169Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:56.5037794Z ##[debug]arguments:
2022-05-30T05:03:56.5038411Z ##[debug]arguments:
2022-05-30T05:03:56.5039001Z ##[debug] login
2022-05-30T05:03:56.5039589Z ##[debug] login
2022-05-30T05:03:56.5040324Z ##[debug] --service-principal
2022-05-30T05:03:56.5041144Z ##[debug] --service-principal
2022-05-30T05:03:56.5041849Z ##[debug] -u
2022-05-30T05:03:56.5042524Z ##[debug] -u
2022-05-30T05:03:56.5043286Z ##[debug] ***
2022-05-30T05:03:56.5044061Z ##[debug] ***
2022-05-30T05:03:56.5044930Z ##[debug] --password=***
2022-05-30T05:03:56.5045830Z ##[debug] --password=***
2022-05-30T05:03:56.5046551Z ##[debug] --tenant
2022-05-30T05:03:56.5047237Z ##[debug] --tenant
2022-05-30T05:03:56.5047996Z ##[debug] 9ad0e722-d594-42e3-8fd3-34389ad0b720
2022-05-30T05:03:56.5048818Z ##[debug] 9ad0e722-d594-42e3-8fd3-34389ad0b720
2022-05-30T05:03:56.5049619Z ##[debug] --allow-no-subscriptions
2022-05-30T05:03:56.5050379Z ##[debug] --allow-no-subscriptions
2022-05-30T05:03:56.5051691Z [command]/usr/bin/az login --service-principal -u *** --password=*** --tenant 9ad0e722-d594-42e3-8fd3-34389ad0b720 --allow-no-subscriptions
2022-05-30T05:03:58.0753666Z [
2022-05-30T05:03:58.0754649Z {
2022-05-30T05:03:58.0755172Z "cloudName": "AzureCloud",
2022-05-30T05:03:58.0756618Z "homeTenantId": "9ad0e722-d594-42e3-8fd3-34389ad0b720",
2022-05-30T05:03:58.0757534Z "id": "195ae8df-2d72-46f3-8d91-e83a1401efd2",
2022-05-30T05:03:58.0758136Z "isDefault": true,
2022-05-30T05:03:58.0758667Z "managedByTenants": [],
2022-05-30T05:03:58.0759372Z "name": "Pay-As-You-Go",
2022-05-30T05:03:58.0759925Z "state": "Enabled",
2022-05-30T05:03:58.0760698Z "tenantId": "9ad0e722-d594-42e3-8fd3-34389ad0b720",
2022-05-30T05:03:58.0761287Z "user": {
2022-05-30T05:03:58.0762698Z "name": "***",
2022-05-30T05:03:58.0763277Z "type": "servicePrincipal"
2022-05-30T05:03:58.0763788Z }
2022-05-30T05:03:58.0764240Z }
2022-05-30T05:03:58.0764686Z ]
2022-05-30T05:03:58.0765888Z ##[debug]which 'az'
2022-05-30T05:03:58.0766921Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:03:58.0768080Z ##[debug]/usr/bin/az arg: account set --subscription "195ae8df-2d72-46f3-8d91-e83a1401efd2"
2022-05-30T05:03:58.0769371Z ##[debug]/usr/bin/az arg: account set --subscription "195ae8df-2d72-46f3-8d91-e83a1401efd2"
2022-05-30T05:03:58.0771527Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:58.0772412Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:03:58.0773202Z ##[debug]arguments:
2022-05-30T05:03:58.0773954Z ##[debug]arguments:
2022-05-30T05:03:58.0775074Z ##[debug] account
2022-05-30T05:03:58.0775829Z ##[debug] account
2022-05-30T05:03:58.0776558Z ##[debug] set
2022-05-30T05:03:58.0777291Z ##[debug] set
2022-05-30T05:03:58.0778256Z ##[debug] --subscription
2022-05-30T05:03:58.0779169Z ##[debug] --subscription
2022-05-30T05:03:58.0780128Z ##[debug] 195ae8df-2d72-46f3-8d91-e83a1401efd2
2022-05-30T05:03:58.0781115Z ##[debug] 195ae8df-2d72-46f3-8d91-e83a1401efd2
2022-05-30T05:03:58.0782110Z [command]/usr/bin/az account set --subscription 195ae8df-2d72-46f3-8d91-e83a1401efd2
2022-05-30T05:03:58.3479742Z ##[debug]addSpnToEnvironment=false
2022-05-30T05:03:58.3481939Z ##[debug]exec tool: /usr/bin/bash
2022-05-30T05:03:58.3482812Z ##[debug]arguments:
2022-05-30T05:03:58.3483724Z ##[debug] /home/vsts/work/1/s/tools/utils/create_pull_request.sh
2022-05-30T05:03:58.3485276Z ##[debug] --organization-url
2022-05-30T05:03:58.3486132Z ##[debug] https://dev.azure.com/naganu836/
2022-05-30T05:03:58.3487092Z ##[debug] --project-name
2022-05-30T05:03:58.3487985Z ##[debug] williams-poc
2022-05-30T05:03:58.3490235Z ##[debug] --repository-name
2022-05-30T05:03:58.3491448Z ##[debug] https://[email protected]/naganu836/williams-poc/_git/apiops.git
2022-05-30T05:03:58.3492914Z ##[debug] --pull-request-title
2022-05-30T05:03:58.3493794Z ##[debug] Merging artifacts from portal (Build 180)
2022-05-30T05:03:58.3495109Z ##[debug] --branch-name
2022-05-30T05:03:58.3495884Z ##[debug] main
2022-05-30T05:03:58.3496765Z ##[debug] --source-folder-path
2022-05-30T05:03:58.3497715Z ##[debug] /home/vsts/work/1/artifacts-from-portal
2022-05-30T05:03:58.3498704Z ##[debug] --temporary-branch-name
2022-05-30T05:03:58.3499663Z ##[debug] artifacts-from-portal-build-180
2022-05-30T05:03:58.3500614Z ##[debug] --temporary-folder-path
2022-05-30T05:03:58.3501576Z ##[debug] /home/vsts/work/_temp/artifacts-from-portal
2022-05-30T05:03:58.3503365Z ##[debug] --overwrite-subfolder
2022-05-30T05:03:58.3504009Z ##[debug] extract
2022-05-30T05:03:58.3505889Z [command]/usr/bin/bash /home/vsts/work/1/s/tools/utils/create_pull_request.sh --organization-url https://dev.azure.com/naganu836/ --project-name williams-poc --repository-name https://[email protected]/naganu836/williams-poc/_git/apiops.git --pull-request-title Merging artifacts from portal (Build 180) --branch-name main --source-folder-path /home/vsts/work/1/artifacts-from-portal --temporary-branch-name artifacts-from-portal-build-180 --temporary-folder-path /home/vsts/work/_temp/artifacts-from-portal --overwrite-subfolder extract
2022-05-30T05:03:58.3546457Z Installing Azure DevOps extension...
2022-05-30T05:03:58.5711113Z WARNING: Extension 'azure-devops' is already installed.
2022-05-30T05:03:59.1330024Z Creating folder /home/vsts/work/_temp/artifacts-from-portal...
2022-05-30T05:03:59.1343211Z Cloning branch main...
2022-05-30T05:04:00.9118371Z ERROR: A potentially dangerous Request.Path value was detected from the client (:).
2022-05-30T05:04:01.0136986Z ##[debug]Exit code 1 received from tool '/usr/bin/bash'
2022-05-30T05:04:01.0137863Z ##[debug]STDIO streams have closed for tool '/usr/bin/bash'
2022-05-30T05:04:01.0138494Z ##[debug]task result: Failed
2022-05-30T05:04:01.0167923Z ##[error]Script failed with exit code: 1
2022-05-30T05:04:01.0177990Z ##[debug]Processed: ##vso[task.issue type=error;]Script failed with exit code: 1
2022-05-30T05:04:01.0185093Z ##[debug]Processed: ##vso[task.complete result=Failed;]Script failed with exit code: 1
2022-05-30T05:04:01.0186431Z ##[debug]which 'az'
2022-05-30T05:04:01.0187106Z ##[debug]found: '/usr/bin/az'
2022-05-30T05:04:01.0187687Z ##[debug]/usr/bin/az arg: account clear
2022-05-30T05:04:01.0188301Z ##[debug]/usr/bin/az arg: account clear
2022-05-30T05:04:01.0188889Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:04:01.0189442Z ##[debug]exec tool: /usr/bin/az
2022-05-30T05:04:01.0189992Z ##[debug]arguments:
2022-05-30T05:04:01.0190510Z ##[debug]arguments:
2022-05-30T05:04:01.0191020Z ##[debug] account
2022-05-30T05:04:01.0191542Z ##[debug] account
2022-05-30T05:04:01.0192047Z ##[debug] clear
2022-05-30T05:04:01.0192548Z ##[debug] clear
2022-05-30T05:04:01.0192908Z [command]/usr/bin/az account clear
2022-05-30T05:04:01.3016567Z ##[section]Finishing: Create pull request with artifacts