Connection String question about durabletask-mssql

[SeaDude]

Anyone have thoughts on this one?

[SeaDude]

I'd like to use this tool to track Azure Durable Function state.

[cgillum]

Hi @SeaDude. I haven't tried this exact scenario when using Azure Managed Identities, but the intent is that you specify the name of the database that you want to be created in your connection string. When the app starts up, it will first connect to the master database and will then create the target database (from the connection string) if it doesn't yet exist. Have you tried this yet, and what was the outcome?

[SeaDude]

Thank you for the suggestion. The question I have is, what is the Connection String to use? I'm using Azure SQL and there doesn't seem to be a default ConnString to use to access the server/master database.

[cgillum]

Does this help? https://learn.microsoft.com/en-us/azure/azure-sql/database/connect-query-content-reference-guide?view=azuresql#get-adonet-connection-information-optional---sql-database-only. Just replace the server name with your server name and the database name with master?

[SeaDude]

@cgillum

1. I added SQLDB_Connection: "Server=tcp:<my-server>.database.windows.net,1433;Initial Catalog=sqldb-durable-function;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Authentication=Active Directory Managed Identity;" connection string to local.settings.json.
2. I assigned the Contributor RBAC role to the Function App's System-Assigned Managed Identity (SAMI)
3. Then published the local Function to the Azure Function App

I'm not sure where to go from there. There was nothing in the func publish logs stating that the database was being built. Remote build succeeded! is what I received and the Function is present in Azure.

Do you know how I should debug the auto-db creation?

[cgillum]

Database creation should be happening when the app starts up for the first time, not during deployment. Is your app starting up successfully? If not, are there any errors in the logs that might indicate whether a problem occurred?

[SeaDude]

Hello @cgillum,

I the Function has been deployed to the Function App. I triggered the app, and now I get an exception loop...

Here is the App Setting that is deployed:

• 
• Server=tcp:<my-server>.database.windows.net,1433;Initial Catalog=sqldb-durable-function;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Authentication=Active Directory Managed Identity;

Exception that is pulled from Azure Function Live Metrics:

• Initial exception:

Time	8:53:24 PM
Exception type	Microsoft.Azure.WebJobs.Host.FunctionInvocationException
Exception message	Exception while executing function: Functions.trigger_function <--- Result: Failure Exception: Exception: {'Message': 'Something went wrong while processing your request', 'ExceptionMessage': "Login failed for user '<token-identified principal>'.", 'ExceptionType': 'Microsoft.Data.SqlClient.SqlException', 'StackTrace': ' at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)\n at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()\n--- End of stack trace from previous location ---\n at DurableTask.SqlServer.SqlOrchestrationService.GetAndOpenConnectionAsync(CancellationToken cancelToken) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 85\n at DurableTask.SqlServer.SqlOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 503\n at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 614\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\ContextImplementations\\DurableClient.cs:line 210\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleStartOrchestratorRequestAsync(HttpRequestMessage request, String functionName, String instanceId) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 875\n at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleRequestAsync(HttpRequestMessage request) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 306'} Stack: File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 489, in _handle__invocation_request call_result = await self._run_async_func( File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 768, in _run_async_func return await ExtensionManager.get_async_invocation_wrapper( File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/extension.py", line 147, in get_async_invocation_wrapper result = await function(**args) File "/home/site/wwwroot/trigger_function/__init__.py", line 9, in main instance_id = await durable_orc_client.start_new(req.route_params["functionName"], None, None) File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 89, in start_new raise Exception(ex_message)
Category	Host.Results
HostInstanceId	5bf59404-e91a-4b23-a719-2d27016ee832
InvocationId	762cb8f2-a953-42f3-a2d4-0943e8d6d52c
LogLevel	Error
ProcessId	24
prop__Duration	00:00:18.2652687
prop__EndTime	2023-11-17T04:53:23.956Z
prop__FullName	Functions.trigger_function
prop__InvocationId	762cb8f2-a953-42f3-a2d4-0943e8d6d52c
prop__Name	trigger_function
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.trigger_function
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: Exception: {'Message': 'Something went wrong while processing your request', 'ExceptionMessage': "Login failed for user '<token-identified principal>'.", 'ExceptionType': 'Microsoft.Data.SqlClient.SqlException', 'StackTrace': '   at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.WaitForPendingOpen()\n--- End of stack trace from previous location ---\n   at DurableTask.SqlServer.SqlOrchestrationService.GetAndOpenConnectionAsync(CancellationToken cancelToken) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 85\n   at DurableTask.SqlServer.SqlOrchestrationService.CreateTaskOrchestrationAsync(TaskMessage creationMessage, OrchestrationStatus[] dedupeStatuses) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 503\n   at DurableTask.Core.TaskHubClient.InternalCreateOrchestrationInstanceWithRaisedEventAsync(String orchestrationName, String orchestrationVersion, String orchestrationInstanceId, Object orchestrationInput, IDictionary`2 orchestrationTags, OrchestrationStatus[] dedupeStatuses, String eventName, Object eventData, Nullable`1 startAt) in /_/src/DurableTask.Core/TaskHubClient.cs:line 614\n   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableClient.Microsoft.Azure.WebJobs.Extensions.DurableTask.IDurableOrchestrationClient.StartNewAsync[T](String orchestratorFunctionName, String instanceId, T input) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\ContextImplementations\\DurableClient.cs:line 210\n   at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleStartOrchestratorRequestAsync(HttpRequestMessage request, String functionName, String instanceId) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 875\n   at Microsoft.Azure.WebJobs.Extensions.DurableTask.HttpApiHandler.HandleRequestAsync(HttpRequestMessage request) in D:\\a\\_work\\1\\s\\src\\WebJobs.Extensions.DurableTask\\HttpApiHandler.cs:line 306'}
Stack:   File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 489, in _handle__invocation_request
    call_result = await self._run_async_func(
  File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 768, in _run_async_func
    return await ExtensionManager.get_async_invocation_wrapper(
  File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/extension.py", line 147, in get_async_invocation_wrapper
    result = await function(**args)
  File "/home/site/wwwroot/trigger_function/__init__.py", line 9, in main
    instance_id = await durable_orc_client.start_new(req.route_params["functionName"], None, None)
  File "/home/site/wwwroot/.python_packages/lib/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 89, in start_new
    raise Exception(ex_message)

   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 108

Recurring exception (every few minutes):

Time	9:25:31 PM
Exception type	Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException
Exception message	The listener for function 'Functions.orchestrator' was unable to start. <--- CREATE DATABASE permission denied in database 'master'.
Category	Host.Startup
FormattedMessage	The listener for function 'Functions.orchestrator' was unable to start.
HostInstanceId	5bf59404-e91a-4b23-a719-2d27016ee832
LogLevel	Error
ProcessId	24
prop__{OriginalFormat}	The listener for function 'Functions.orchestrator' was unable to start.
Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'Functions.orchestrator' was unable to start.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): CREATE DATABASE permission denied in database 'master'.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 497
   at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 504
   at DurableTask.SqlServer.SqlUtils.ExecuteSprocAndTraceAsync[T](DbCommand command, LogHelper traceHelper, String instanceId, Func`2 executor) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 439
   at DurableTask.SqlServer.SqlDbManager.CreateDatabaseAsync(String databaseName, SqlConnection connection) in /_/src/DurableTask.SqlServer/SqlDbManager.cs:line 208
   at DurableTask.SqlServer.SqlDbManager.EnsureDatabaseExistsAsync() in /_/src/DurableTask.SqlServer/SqlDbManager.cs:line 187
   at DurableTask.SqlServer.SqlDbManager.AcquireDatabaseLockAsync(Boolean createDatabaseIfNotExists) in /_/src/DurableTask.SqlServer/SqlDbManager.cs:line 137
   at DurableTask.SqlServer.SqlDbManager.CreateOrUpgradeSchemaAsync(Boolean recreateIfExists) in /_/src/DurableTask.SqlServer/SqlDbManager.cs:line 33
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.StartTaskHubWorkerIfNotStartedAsync() in D:\a\_work\1\s\src\WebJobs.Extensions.DurableTask\DurableTaskExtension.cs:line 1429
   at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 68
ClientConnectionId:f837b601-820e-4cb4-b6f4-b6ee649f261f
Error Number:262,State:1,Class:14
ClientConnectionId before routing:89436671-2a21-4d2e-aeff-72557ef2ab00
Routing Destination:cbe7cd08f62a.tr11103.westus2-a.worker.database.windows.net,11000
   --- End of inner exception stack trace ---

I've already created the Managed Identity USER in the master database...

CREATE USER [nameofapplication] FROM EXTERNAL PROVIDER;

...but each time I try to add the USER to the db_owner role, I get the error shown, even though I'm logged in as the SQL Admin. I also tried logging into the server as the SQL AAD Admin. Same error:

ALTER ROLE db_owner ADD MEMBER [nameofapplication];

Msg 15151, Level 16, State 1, Line 1
Cannot alter the role 'db_owner', because it does not exist or you do not have permission.

[cgillum]

Hi @SeaDude - thanks for providing all this information. Based on the error messages you're seeing, the issue seems to be related to permissions. Does anything in the following documentation provide hints about what might be wrong? https://learn.microsoft.com/en-us/sql/t-sql/statements/create-database-transact-sql?view=azuresqldb-current&preserve-view=true&tabs=sqlpool#permissions-1. This part seems relevant:

To create a database, the user login must be one of the following principals:

• The server-level principal login
• The Azure AD administrator for the local Azure SQL Server
• A login that is a member of the dbmanager database role

[SeaDude]

Thanks for the continued engagement @cgillum. I was able to successfully execute ALTER ROLE dbmanager ADD MEMBER [my-managed-identity]. Looks like the database is now being created!!

[SeaDude]

I think this part of the docs need to be updated (maybe?) to read dbmanager instead of db_owner.

Thanks again for the assistance!

Nice!

[SeaDude]

Don't forget to change the db Compute + Storage! The auto-deploy creates a General Purpose instance by default. Changed to the Basic instance for development.