[BUG] DefaultAzureCredential does not fall back to other credential types unless ManagedIdentityCredential is disabled about azure-sdk-for-net

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[christothes]

Hi @jeppe9821 -
Would you mind providing the logging output after reproducing this with logging enabled?

[github-actions]

Hi @jeppe9821. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jeppe9821]

@christothes Here is the exception occuring when IsContentLogging is set to 'true'

An unhandled exception of type 'Azure.Identity.AuthenticationFailedException' occurred in Azure.Security.KeyVault.Secrets.dll
ManagedIdentityCredential authentication failed: Service request failed.
Status: 503 (Service Unavailable)

Content:
{"error":"service_unavailable","error_description":"Service not available, possibly because the machine is not connected to Azure or the config file is missing. Error: missing required agent config properties. Current agent config: {Subscriptionid: Resourcegroup: Resourcename: Tenantid: Location: VMID: VMUUID: CertificateThumbprint: Clientid: Cloud: PrivateLinkScope: Namespace: CorrelationID: ArmEndpoint: AtsResourceId:} (config file location: C:\\ProgramData\\AzureConnectedMachineAgent\\Config\\agentconfig.json). Connection status: Disconnected. Check Agent log for more details.","error_codes":[503],"timestamp":"2024-03-13 09:31:16.2211899 +0100 CET m=+64594.598745201","trace_id":"","correlation_id":"7d009a43-de9d-474c-9d15-8bef2611b0f1"}

Call stack:

 Azure.Identity.AuthenticationFailedException
  HResult=0x80131500
  Message=ManagedIdentityCredential authentication failed: Service request failed.
Status: 503 (Service Unavailable)

Content:
{"error":"service_unavailable","error_description":"Service not available, possibly because the machine is not connected to Azure or the config file is missing. Error: missing required agent config properties. Current agent config: {Subscriptionid: Resourcegroup: Resourcename: Tenantid: Location: VMID: VMUUID: CertificateThumbprint: Clientid: Cloud: PrivateLinkScope: Namespace: CorrelationID: ArmEndpoint: AtsResourceId:} (config file location: C:\\ProgramData\\AzureConnectedMachineAgent\\Config\\agentconfig.json). Connection status: Disconnected. Check Agent log for more details.","error_codes":[503],"timestamp":"2024-03-13 09:31:16.2211899 +0100 CET m=+64594.598745201","trace_id":"","correlation_id":"7d009a43-de9d-474c-9d15-8bef2611b0f1"}


Headers:
Date: Wed, 13 Mar 2024 08:31:16 GMT
Content-Length: 750
Content-Type: text/plain; charset=utf-8

See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot
  Source=Azure.Identity
  StackTrace:
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.ManagedIdentityCredential.<GetTokenImplAsync>d__16.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
   at Azure.Identity.ManagedIdentityCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Identity.DefaultAzureCredential.<GetTokenFromSourcesAsync>d__14.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
   at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex, String additionalMessage, Boolean isCredentialUnavailable)
   at Azure.Identity.DefaultAzureCredential.<GetTokenImplAsync>d__12.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
   at Azure.Identity.DefaultAzureCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueFromCredentialAsync>d__9.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](Task`1 task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AccessTokenCache.<GetHeaderValueAsync>d__6.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.AuthenticateAndAuthorizeRequest(HttpMessage message, TokenRequestContext context)
   at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthorizeRequestOnChallengeAsyncInternal>d__10.MoveNext()
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted[T](ValueTask`1 task)
   at Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthorizeRequestOnChallenge(HttpMessage message)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.<ProcessAsync>d__11.MoveNext()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.BearerTokenAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__7.MoveNext()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.RedirectPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Azure.Core.Pipeline.TaskExtensions.EnsureCompleted(ValueTask task)
   at Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken)
   at Azure.Core.Pipeline.HttpPipeline.SendRequest(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.KeyVaultPipeline.GetPage[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken)
   at Azure.Security.KeyVault.Secrets.SecretClient.<>c__DisplayClass15_0.<GetPropertiesOfSecrets>b__0(String nextLink)
   at Azure.Core.PageResponseEnumerator.<>c__DisplayClass0_0`1.<CreateEnumerable>b__0(String continuationToken, Nullable`1 pageSizeHint)
   at Azure.Core.PageResponseEnumerator.FuncPageable`1.<AsPages>d__2.MoveNext()
   at Azure.Pageable`1.<GetEnumerator>d__8.MoveNext()
   at Azure.Extensions.AspNetCore.Configuration.Secrets.AzureKeyVaultConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.InitializeAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at GSOrderAPIAdapter.Program.Main() in C:\Dev\GS-Order-API-Adapter\src\GSOrderAPIAdapter\GSOrderAPIAdapter\Program.cs:line 14

  This exception was originally thrown at this call stack:
    Azure.Identity.ManagedIdentitySource.HandleResponseAsync(bool, Azure.Core.TokenRequestContext, Azure.Response, System.Threading.CancellationToken)
    System.Threading.Tasks.ValueTask<TResult>.Result.get()
    System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<TResult>.ConfiguredValueTaskAwaiter.GetResult()
    Azure.Identity.AzureArcManagedIdentitySource.HandleResponseAsync(bool, Azure.Core.TokenRequestContext, Azure.Response, System.Threading.CancellationToken)
    System.Threading.Tasks.ValueTask<TResult>.Result.get()
    System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<TResult>.ConfiguredValueTaskAwaiter.GetResult()
    Azure.Identity.ManagedIdentitySource.AuthenticateAsync(bool, Azure.Core.TokenRequestContext, System.Threading.CancellationToken)
    System.Threading.Tasks.ValueTask<TResult>.Result.get()
    System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<TResult>.ConfiguredValueTaskAwaiter.GetResult()
    Azure.Identity.ManagedIdentityClient.AuthenticateCoreAsync(bool, Azure.Core.TokenRequestContext, System.Threading.CancellationToken)
    ...
    [Call Stack Truncated]

Inner Exception 1:
RequestFailedException: Service request failed.
Status: 503 (Service Unavailable)

Content:
{"error":"service_unavailable","error_description":"Service not available, possibly because the machine is not connected to Azure or the config file is missing. Error: missing required agent config properties. Current agent config: {Subscriptionid: Resourcegroup: Resourcename: Tenantid: Location: VMID: VMUUID: CertificateThumbprint: Clientid: Cloud: PrivateLinkScope: Namespace: CorrelationID: ArmEndpoint: AtsResourceId:} (config file location: C:\\ProgramData\\AzureConnectedMachineAgent\\Config\\agentconfig.json). Connection status: Disconnected. Check Agent log for more details.","error_codes":[503],"timestamp":"2024-03-13 09:31:16.2211899 +0100 CET m=+64594.598745201","trace_id":"","correlation_id":"7d009a43-de9d-474c-9d15-8bef2611b0f1"}


Headers:
Date: Wed, 13 Mar 2024 08:31:16 GMT
Content-Length: 750
Content-Type: text/plain; charset=utf-8

Code:

public static void Main()
        {
            var host = new HostBuilder()
                .ConfigureFunctionsWebApplication()
                .ConfigureAppConfiguration(ConfigureApp)
                .ConfigureServices(ConfigureServices)                
                .Build();

            host.Run();
        }

        private static void ConfigureApp(HostBuilderContext hostBuilderContext, IConfigurationBuilder configurationBuilder)
        {
            var config = configurationBuilder.Build();

            DefaultAzureCredentialOptions options = new DefaultAzureCredentialOptions
            {
                Diagnostics =
                {
                    LoggedHeaderNames = { "x-ms-request-id" },
                    LoggedQueryParameters = { "api-version" },
                    IsLoggingContentEnabled = true
                }
            };

            configurationBuilder.AddAzureKeyVault(new Uri(config["urikeyvault"]), new DefaultAzureCredential(options));
            
            if (hostBuilderContext.HostingEnvironment.IsDevelopment())
            {
                configurationBuilder.AddUserSecrets<Program>();
            }
        }

        private static void ConfigureServices(IServiceCollection services)
        {
            services.AddHttpClient();
        }```

[christothes]

Hi @jeppe9821 -
From what I can see from the exception is that the IMDS service endpoint is returning a 503 error. Are you running this on your local dev machine or hosted in Azure?

[github-actions]

Hi @jeppe9821. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[jeppe9821]

Hi @christothes , yes I'm running this locally from Visual Studio and when running the program I get a localhost endpoint (eg http://localhost:7071/api/). However I am trying to access the services externally eg the KeyVault is taken from the DEV environment in Azure

The AzureWebJobsStorage is set to UseDevelopmentStorage=true

(full config file)

{ "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "UseDevelopmentStorage=true", "AzureWebJobsSecretStorageType": "files", "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated", "FUNCTIONS_EXTENSION_VERSION": "~4", "KeyVaultUri": "https://.....vault.azure.net/" } }

The thing is though, I have a colleague running the exact same project on a different machine with the exact same configurations and for them it passes. So there's something in my local environment specifically causing this error

[christothes]

Can you share the log output produced from adding an event listener as described in the logging link above?

ex:

// Setup a listener to monitor logged events.
using AzureEventSourceListener listener = AzureEventSourceListener.CreateConsoleLogger();

or if you need to route it through the webjob logger, you can setup a custom listener as described here

[github-actions]

Hi @jeppe9821. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions]

Hi @jeppe9821, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!