RpcException - PurgeAllInstancesAsync operation was canceled - HttpClient.Timeout of 100 seconds elapsing about durabletask-dotnet

[ciranmc]

@CGSK @lilyjma Did we ever get a fix or workaround for this? I am facing exactly the same issue when running Azure Function locally but with the WaitForInstanceCompleteAsync method. Seems like a fundamental issue as these methods are designed for long running processes. There has to be some easy way to configure the httpClient timeout via services collection or something. Any help would be greatly appreciated.

[CGSK]

@lilyjma Timeout only happens if we have plenty of instances to delete which is obviously the case in our prod.

[Function("PurgeDurableFunctionsHistory")]
public async Task RunAsync([TimerTrigger("%PurgeDurableFunctionsHistory:Schedule%")] TimerInfo myTimer, [DurableClient] DurableTaskClient client, CancellationToken cancellationToken)
{
PurgeResult result = null;
if (!int.TryParse(Environment.GetEnvironmentVariable("PurgeDurableFunctionsHistoryDays"), out int deletePurgeHistoryDays))
{
deletePurgeHistoryDays = 30;
}

var orchStatus = new List<OrchestrationRuntimeStatus>
{
	OrchestrationRuntimeStatus.Completed,
};
var purgeFilter = new PurgeInstancesFilter(
	DateTime.SpecifyKind(DateTime.MinValue, DateTimeKind.Utc),
	DateTime.SpecifyKind(DateTime.UtcNow.AddDays(deletePurgeHistoryDays * -1), DateTimeKind.Utc),
	orchStatus);

result = await client.PurgeAllInstancesAsync(purgeFilter, cancellationToken).ConfigureAwait(false);

}

[lilyjma]

@CGSK - could you please provide a simple sample for reproducing this?

[lilyjma]

@CGSK - Could you try the following?

Rather than increasing the timeout, try and split purging into multiple steps, making each step small enough to succeed. For example, purge data older than 12 months, then purge data older than 11 months, eventually reaching the 30 days target. You may need to adjust the numbers. This workaround is more scalable than increasing the timeout.

[CGSK]

@lilyjma In our case, we started a new task hub with just 7 days old data, the purge is failing with timeout. Is there any way we can configure the http or function execution timeout of the actual sidecar process? If PurgeAllInstancesAsync makes a http call from the sidecar process to purge task hub and if the host's timeout configuration is not pushed to sidecar process, this is very constraining. I wonder this sidecar http timeout is specific to DurableTaskClient.PurgeAllInstancesAsync (Grpc) implementation or this is even an issue with any custom http client calls (for ex. using System.Net.Http.HttpClient).

[CGSK]

@lilyjma This is even happening in the latest NuGet version.

<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.1.4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask.SqlServer" Version="1.3.0" />