[BUG] CosmosDBAccountResource.RegenerateKeyAsync throws exception on 202 status code about azure-sdk-for-net

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[archerzz]

@baryoloraul Could you share the codes to invoke RegenerateKeyAsync() and the detailed error messages?

I cannot reproduce this error under Azure.ResourceManager.CosdmosDB version 1.3.2 and Azure.Identity version 1.12.0-beta.2.
Here are my codes (REDACTED):

using AzureEventSourceListener consoleListener = AzureEventSourceListener.CreateConsoleLogger(EventLevel.Informational);
var client = new ArmClient(new InteractiveBrowserCredential(), "{your subscription id}", new ArmClientOptions()
{
    Diagnostics = { IsLoggingContentEnabled = true }
});

ResourceGroupResource rg = client.GetDefaultSubscription().GetResourceGroup("{your resource group}");
CosmosDBAccountResource account = rg.GetCosmosDBAccount("{your cosmos account}");
await account.RegenerateKeyAsync(WaitUntil.Completed, new CosmosDBAccountRegenerateKeyContent(CosmosDBAccountKeyKind.Secondary));

Here is some key traffic content:

[Informational] Azure-Core: Request [xxxxxxxx] POST https://management.azure.com/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.DocumentDB/databaseAccounts/xxxxxx/regenerateKey?api-version=2022-11-15
[Informational] Azure-Core: Response [xxxxxxx] 202 Accepted (01.6s)
......
[Informational] Azure-Core: Request [yyyyy] GET https://management.azure.com/subscriptions/xxx/providers/Microsoft.DocumentDB/locations/eastus2/operationsStatus/xxxxxxxxxxx?api-version=2022-11-15
[Informational] Azure-Core: Response [yyyyy] 200 Ok (00.6s)
......
[Informational] Azure-Core: Request [zzzzz] GET https://management.azure.com/subscriptions/xxx/resourceGroups/xxxx/providers/Microsoft.DocumentDB/databaseAccounts/xxxxxxxx/regenerateKey/operationResults/xxxxxxxxx?api-version=2022-11-15
[Informational] Azure-Core: Response [zzzzz] 200 Ok (00.6s)

Apparently, the polling is successful.

[github-actions]

Hi @baryoloraul. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @baryoloraul, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[baryoloraul]

Hi @archerzz, the error details in the bug description is all I have, next is the code I am using to rotate the keys:

var cosmosDbAccountId = CosmosDBAccountResource.CreateResourceIdentifier(subscriptionId, resourceGroupName, cosmosDbAccountName);
var cosmosDbAccount = armClient.GetCosmosDBAccountResource(cosmosDbAccountId);
var operationResult = await cosmosDbAccount.RegenerateKeyAsync(WaitUntil.Completed, new CosmosDBAccountRegenerateKeyContent(CosmosDBAccountKeyKind.SecondaryReadonly), cancellationToken).ConfigureAwait(false);

Worth mentioning that this error does not show up every time I try to rotate the keys, I haven't found a pattern though. Retrying the same failed operation succeeds some times.

I am running this in .Net 6 in CBL-Mariner 2.0.

[archerzz]

I compared our test recording and found there could be a status problem on backend.

• See our test recording of regenerating keys: https://github.com/Azure/azure-sdk-assets/blob/6f2e7f0c2ec8320d22226026a9055ccc9f14749a/net/sdk/cosmosdb/Azure.ResourceManager.CosmosDB/tests/SessionRecords/DatabaseAccountTests/DatabaseAccountListKeysAndRegenerateKeysTest.json#L866 It returns "status": "Enqueued" for the 1st request.
• But from the error diagnostics given in this issue, the initial response body is {"status":"Dequeued"}.
• I checked the test recordings and found that normally the status should be Enqueued, then Dequeued, then Succeeded. So, I'm wondering if there is something wrong on backend. That could also explain why this issue is intermittent.

Hi cosmos team,
Can you check x-ms-request-id: a7d25542-9579-44be-882b-4fc4a265c5be to see if there is something wrong on backend? Thanks.

[archerzz]

@baryoloraul Is it possible you enable diagnostics on your codes? like below:

// or other listener to collect logs
using AzureEventSourceListener consoleListener = AzureEventSourceListener.CreateConsoleLogger(EventLevel.Informational);

var client = new ArmClient(new AzureDefaultCredential(), "xxxx", new ArmClientOptions()
{
    Diagnostics = { IsLoggingContentEnabled = true }
});

We cannot reproduce this error. And after analyzing existing error messages, there is no clue on what was wrong. Some detailed logs are greatly appreciated. Thanks.

[github-actions]

Hi @baryoloraul. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @baryoloraul, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[github-actions]

Hi @baryoloraul, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!