[BUG] Error while trying to access keys stored in Azure KeyVault about azure-sdk-for-net

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

Hi @ekarim2. Thanks for reaching out and we regret that you're experiencing difficulties. The error message that you're seeing does not appear to be related to KeyVault. Rather it seems to indicate that the local host could not find an encryption key to decrypt data.

Please provide the full error message and stack trace for the error as well as information about what host platform your app is running on.

[github-actions]

Hi @ekarim2. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[heaths]

We do try to cache the key locally which may be added to an in-memory key ring depending on your platform. Caching the key reduces not only latency but decreases the cost for customers since Key Vault has low rate limits. Still, why you're getting this particular message isn't clear.

What OS is this running on? Can you enable Open Telemetry and provide logs showing when this particular option occurred? There should be tracing events like a GET /keys/{key-name} event at least.

Forcing remote-only isn't something we support currently but are considering. /cc @vcolin7 @nisha-bhatia

[ekarim2]

Error Messages:
One or more errors occurred. (ClientSecret couldn't be fetched from KeyVault. Access Token couldn't be fetched for further processing). A task was canceled.

Exception while executing function: -------- ClientSecret couldn't be fetched from KeyVault. Access Token couldn't be fetched for further processing. One or more errors occurred. A task was canceled.

One or more error occurred. (ClientSecret couldn't be fetched from KeyVault, Access Token couldn't be fetched for further processing). An error occurred while sending the request. Unable to read data from transport connection: An existing connection was forcibly closed by the remote host.

Note:

1. The error was first observed when there was a KeyVault (Azure) outage in Azure West-Europe region. Our services are also hosted in the same region.
2. The error was observed again when there was issues of Azure authentication services in third week of May.

The OS is Windows Server 2022.

Thanks

[jsquire]

@ekarim2: We are still missing some important context that was asked for. Given that this was an aggregate exception, please provide the full error message and stack trace for each error in the set.

[github-actions]

Hi @ekarim2. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[ekarim2]

Stack Trace of two errors:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=----)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=----)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=---: \s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)
Inner exception System.Exception handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at ---.ExceptionHandler+<HandleExceptionAsync>d__5.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: \s\---\Handler\ExceptionHandler.cs:153)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---.Handler.ExceptionHandler+<HandleExceptions>d__4.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: --\s\Handler\ExceptionHandler.cs:112)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---ServiceBusTrigger+<Run>d__7.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: ---\---ServiceBusTrigger.cs:74)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=---: ---\src\Microsoft.Azure.WebJobs.Host\Executors\VoidTaskMethodInvoker.cs:20)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=---: ---\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:52)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=---: \src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:581)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=---: s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=--)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.39.0, Culture=neutral, PublicKeyToken=3---: \src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)
Inner exception System.AggregateException handled at ---.Handler.ExceptionHandler+<HandleExceptionAsync>d__5.MoveNext:
   at System.Threading.Tasks.Task.ThrowIfExceptional (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Threading.Tasks.Task`1.GetResultCore (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---KeyVault.Handler.KeyVaultHandler.GetSecretBundle (F---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
   at ---.KeyVault.AuthTokenService+<GetAccessToken>d__7.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Threading.Tasks.TaskCanceledException handled at System.Threading.Tasks.Task.ThrowIfExceptional:

System.AggregateException:
   at System.Threading.Tasks.Task.ThrowIfExceptional (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Threading.Tasks.Task`1.GetResultCore (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---ServiceBusTrigger.GetServiceToken (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: sServiceBusTrigger.cs:86)
   at ---ServiceBusTrigger+<Process>d__9.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: ---ServiceBusTrigger.cs:91)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=--)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---ServiceBusTrigger+<Run>d__7.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: ---ServiceBusTrigger.cs:69)
Inner exception System.Exception handled at System.Threading.Tasks.Task.ThrowIfExceptional:
   at ---.AuthTokenService+<GetAccessToken>d__7.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.AggregateException handled at ---.KeyVault.AuthTokenService+<GetAccessToken>d__7.MoveNext:
   at System.Threading.Tasks.Task.ThrowIfExceptional (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Threading.Tasks.Task`1.GetResultCore (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at ---.KeyVault.Handler.KeyVaultHandler.GetSecretBundle (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
   at ---.Core.KeyVault.AuthTokenService+<GetAccessToken>d__7.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.Net.Http.HttpRequestException handled at System.Threading.Tasks.Task.ThrowIfExceptional:
   at Microsoft.Rest.RetryDelegatingHandler+<SendAsync>d__15.MoveNext (Microsoft.Rest.ClientRuntime, Version=2.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Net.Http.HttpClient+<<SendAsync>g__Core|83_0>d.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=----)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.KeyVault.KeyVaultClient+<GetSecretWithHttpMessagesAsync>d__66.MoveNext (Microsoft.Azure.KeyVault, Version=3.0.5.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at Microsoft.Azure.KeyVault.KeyVaultClientExtensions+<GetSecretAsync>d__12.MoveNext (Microsoft.Azure.KeyVault, Version=3.0.5.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at ---.KeyVault.Handler.KeyVaultHandler+<GetSecretBundleAsync>d__10.MoveNext (---, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
Inner exception System.IO.IOException handled at Microsoft.Rest.RetryDelegatingHandler+<SendAsync>d__15.MoveNext:
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException (System.Net.Sockets, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult (System.Net.Sockets, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Net.Security.SslStream+<ReadAsyncInternal>d__188`1.MoveNext (System.Net.Security, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Net.Http.HttpConnection+<<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|45_0>d.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1+ConfiguredValueTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
   at System.Net.Http.HttpConnection+<SendAsyncCore>d__64.MoveNext (System.Net.Http, Version=6.0.0.0, Culture=neutral, PublicKeyToken=---)
Inner exception System.Net.Sockets.SocketException handled at System.Net.Sockets.Socket+AwaitableSocketAsyncEventArgs.ThrowException:

[jsquire]

@ekarim2: The stack trace does not agree with the package version in the summary. In the stack trace, you can see references to:

• Microsoft.Azure.KeyVault.KeyVaultClient
• Microsoft.Rest.ClientRuntime

These indicate that the legacy Microsoft.Azure.KeyVault package is in use, which is not used by Azure.Extensions.AspNetCore.DataProtection.Keys and was never part of its dependency chain. It was, however, used by the legacy Microsoft.AspNetCore.DataProtection.AzureKeyVault package.

Also of note, version 1.1 of Azure.Extensions.AspNetCore.DataProtection.Keys is nearly 3 years old and uses and older build of Azure.Security.KeyVault.Keys which is also ~3 years old. Both are missing a number of important fixes.

At this point, I'm not clear on what the end-to-end scenario is nor how we would be able to reproduce the behavior that you're seeing. In order to assist, we'll need to ask that you share a small, self-contained project that reproduces the behavior that you're seeing with the latest extensions package. That will allows us to take a deeper look and analyze the behavior.

[github-actions]

Hi @ekarim2. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[heaths]

Microsoft.Azure.KeyVault is also deprecated and unsupported, as would be anything that uses it. Please upgrade to the latest Azure.Extensions.AspNetCore.DataProtection.Keys.

[ekarim2]

Thanks all for the support.
We are updating the Azure.Extensions.AspNetCore.DataProtection.Keys version to latest stable version. We will also remove the dependency on Microsoft.Azure.KeyVault and observe if the issue gets resolved. I will update this thread.

[github-actions]

Hi @ekarim2. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @ekarim2, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[ekarim2]

To update all and conclude this issue thread, after version upgrade, we have not observed any error so far. We will continue to monitor for some more time.
Thanks all for the support.