SqlException Transaction (Process ID 66) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. about durabletask-mssql

[yehiyam]

[yehiyam]

Kind ping
@cgillum

[cgillum]

We should be automatically retrying deadlocks, so I wonder if those retries are being exhausted. Unfortunately, it's hard to fix these without having a reliable repro that can be used for testing.

Are you comfortable making changes directly to your database? One thing you can try is changing the deadlock priority of the dt.QuerySingleOrchestration stored procedure, adding a SET DEADLOCK_PRIORITY HIGH to the top of the procedure to make it more likely to survive deadlocks.

For example, you'd need to do an ALTER PROCEDURE statement similar to the following:

ALTER PROCEDURE [dt].[QuerySingleOrchestration]
    @InstanceID varchar(100),
    @ExecutionID varchar(50) = NULL,
    @FetchInput bit = 1,
    @FetchOutput bit = 1
AS
BEGIN
    SET DEADLOCK_PRIORITY HIGH

    DECLARE @TaskHub varchar(50) = dt.CurrentTaskHub()

    SELECT TOP 1
        I.[InstanceID],
        I.[ExecutionID],
        I.[Name],
        I.[Version],
        I.[CreatedTime],
        I.[LastUpdatedTime],
        I.[CompletedTime],
        I.[RuntimeStatus],
        I.[ParentInstanceID],
        (SELECT TOP 1 [Text] FROM Payloads P WHERE
            P.[TaskHub] = @TaskHub AND
            P.[InstanceID] = I.[InstanceID] AND
            P.[PayloadID] = I.[CustomStatusPayloadID]) AS [CustomStatusText],
        CASE WHEN @FetchInput = 1 THEN (SELECT TOP 1 [Text] FROM Payloads P WHERE
            P.[TaskHub] = @TaskHub AND
            P.[InstanceID] = I.[InstanceID] AND
            P.[PayloadID] = I.[InputPayloadID]) ELSE NULL END AS [InputText],
        CASE WHEN @FetchOutput = 1 THEN (SELECT TOP 1 [Text] FROM Payloads P WHERE
            P.[TaskHub] = @TaskHub AND
            P.[InstanceID] = I.[InstanceID] AND
            P.[PayloadID] = I.[OutputPayloadID]) ELSE NULL END AS [OutputText],
        I.[TraceContext]
    FROM Instances I
    WHERE
        I.[TaskHub] = @TaskHub AND
        I.[InstanceID] = @InstanceID AND
        (@ExecutionID IS NULL OR @ExecutionID = I.ExecutionID)
END

If you're comfortable doing so, can you try and see if that helps?

It doesn't fix the deadlock, but it makes it less likely to impact the ReadEntityStateAsync. Instead, the other deadlock participant will get rolled back. Assuming the other deadlock participant is less important or is a background task that can be automatically retried, this could improve the overall availability of your app. However, it could also just move the problem elsewhere.

If you're able to find a way to reproduce this consistently, I could use that reproducer to find a more reliable/permanent fix.

[yehiyam]

Hi @cgillum
I manually applied your change to the stored procedure and it seems to help. After the fix, I wasn't able to reproduce the error.
Is there a way to apply it programmatically when the process starts?
Currently, we are letting the SDK create the DB with the flag task.StorageProvider["createDatabaseIfNotExists"] = true;