Comments (5)
Can you clarify what 1.0.0-rc
refers to? We don't have a Durable Task version of that number. Perhaps you're referring to a nuget package version, such as Microsoft.DurableTask.SqlServer
?
Assuming so, that version is quite old and there may be bugs. It is recommended to upgrade to the latest version, such as 1.2.3
, which also includes important security fixes.
from durabletask-mssql.
Thanks @cgillum . Yes, I was referring the version of SQL Provider nuget package.
I understand it is rather old and we are planning to upgrade to latest version this year, but still we would like to understand the problem better to solve our customer issue.
Do you believe setting the max concurrent activity and orchestration to 1 can resolve the issue?
from durabletask-mssql.
@cgillum Any thought? It would be appreciated if you can provide us some insights of the issue here as we need to unblock out customer installation.
Something we are confused currently:
- Under what circumstance there will be duplicated execution?
- Do you believe set the max concurrent activity and orchestration can resolve the issue?
- If not, how do we ensure the problem does not happen on the latest Microsoft.DurableTask.SqlServer
To provide more clarity. This is the message emitted by DTF MSSQL
2024-04-10 12:46:17.6562|WARN|DurableTask.SqlServer|3d884a6a9fb5498d8c9744f601be1b5f:1: Duplicate execution of 'orchestration' was detected! This means two workers or threads tried to process the same work item simultaneously. The result of this execution will be discarded.
2024-04-10 12:46:17.6577|WARN|DurableTask.SqlServer|3d884a6a9fb5498d8c9744f601be1b5f:1: Duplicate execution of 'orchestration' was detected! This means two workers or threads tried to process the same work item simultaneously. The result of this execution will be discarded.
And then the DT throws this exception under for the same orchestration
2024-04-10 12:46:17.9671|ERROR|DurableTask.Core|TaskActivityDispatcher-c0289b205b9648d09c615d143bf78c4c-0: Unhandled exception with work item '3d884a6a9fb5498d8c9744f601be1b5f:1:0000000000000003': Microsoft.Data.SqlClient.SqlException (0x80131904): The target instance is not running. It may have already completed (in which case this execution can be considered a duplicate) or been terminated. Any results of this task activity execution will be discarded."}
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)"}
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)"}
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)"}
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)"}
at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)"}
at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery(IAsyncResult asyncResult, Boolean isInternal, String endMethod)"}
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)"}
at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync(IAsyncResult asyncResult)"}
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)"}
--- End of stack trace from previous location ---"}
at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 500"}
at DurableTask.SqlServer.SqlUtils.WithRetry[T](Func`1 func, SprocExecutionContext context, LogHelper traceHelper, String instanceId, Int32 maxRetries) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 507"}
at DurableTask.SqlServer.SqlUtils.ExecuteSprocAndTraceAsync[T](DbCommand command, LogHelper traceHelper, String instanceId, Func`2 executor) in /_/src/DurableTask.SqlServer/SqlUtils.cs:line 449"}
at DurableTask.SqlServer.SqlOrchestrationService.CompleteTaskActivityWorkItemAsync(TaskActivityWorkItem workItem, TaskMessage responseMessage) in /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:line 400"}
at DurableTask.Core.TaskActivityDispatcher.OnProcessWorkItemAsync(TaskActivityWorkItem workItem) in /_/src/DurableTask.Core/TaskActivityDispatcher.cs:line 266"}
at DurableTask.Core.TaskActivityDispatcher.OnProcessWorkItemAsync(TaskActivityWorkItem workItem) in /_/src/DurableTask.Core/TaskActivityDispatcher.cs:line 266"}
at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 459"}
ClientConnectionId:8dd9266e-3f22-45f7-8455-0f3b15474870"}
Error Number:50003,State:1,Class:16"}
Backing off for 1 seconds until 5 successful operations"}
from durabletask-mssql.
@jundayin your customer is using a prerelease version of this package, which we don't intend to support.
The cause of these errors is likely related to the worker being overloaded but could also be related to bugs that have been fixed in newer versions of the package. I can't guarantee that you won't see this in a future version.
from durabletask-mssql.
This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.
from durabletask-mssql.
Related Issues (20)
- App fails to start up when upgrading from 1.0.0 to 1.2.0 schema
- Automate database schema upgrade testing HOT 1
- SqlException Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. HOT 4
- Connection String question HOT 12
- NewEvents stuck due to InvalidCastException HOT 6
- MaxActiveOrchestrations set to 1 does not work, mutliple Orchestrations are running HOT 1
- SqlOrchestrationService is not visible when referencing Microsoft.Azure.Functions.Worker.Extensions.DurableTask.SqlServer HOT 1
- Update Microsoft.Data.SqlClient and related dependencies
- Durable Entity creates NullReferenceException with Microsoft.Azure.Functions.Worker.Extensions.DurableTask.SqlServer HOT 5
- Recommendations on storing large payloads HOT 4
- NullReferenceException in SqlORchestrationService HOT 7
- Orchestration execution retry due to OperationCanceledException? HOT 4
- Error propagation is not working properly even after upgrading to latest version of Sql server HOT 1
- Non-Deterministic workflow detected HOT 4
- NU1605: Warning As Error: Detected package downgrade: Microsoft.Azure.WebJobs.Extensions.DurableTask from 2.13.2 to 2.13.1 HOT 1
- Always encrypted support for python HOT 2
- SqlUtils retry logic does not handle DB closed connections HOT 9
- Dt.CreateInstance stored proc is causing deadlocks (proposed ROLLBACK change that fixes the issue) HOT 11
- Change task scheduling to give more even flow of orchestrations
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from durabletask-mssql.