Include child instance IDs in execution history about durabletask-netherite

[sebastianburckhardt]

Thanks for the quick response! Indeed a mystery. Will try to figure it out.

[scale-tone]

Just putting the link to the relevant discussion on DfMon's side: microsoft/DurableFunctionsMonitor#126 (comment)

[sebastianburckhardt]

Instance Ids of suborchestrations are already included as part of the List<History> returned by Netherite. Execution histories are not interpreted or modified in any way by Netherite, this is all just represented exactly like DurableTask.Core defines it.

I think the problem may be that the DFMon implementation is not using the DurableTask.Core representation of histories, which is the same for all storage providers, but a reverse engineering of the Azure Table storage format which is specific to the Azure Table Storage provider?

[scale-tone]

@sebastianburckhardt , DfMon does not do any 'reverse engineering' , when bound to Netherite. For Netherite DfMon falls back to IDurableOrchestrationClient.GetStatusAsync() and renders precisely what that method returns.

Have a test Netherite-based orchestration like this:

Running it locally, with Azurite as a storage and an Event Hub in Azure.

Testing .GetStatusAsync() like this:

        [FunctionName("HttpStatusTest")]
        public static async Task<IActionResult> HttpStatusTest(
            [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
            [DurableClient] IDurableOrchestrationClient starter,
            ILogger log)
        {
            var status = await starter.GetStatusAsync("57f8e7d45e3540ed9e83cfc0f6636812", true, true, true);
            return new ObjectResult(status);
        }

Here is the output:

{
    "name": "DurableFunctionsOrchestrationCSharp1",
    "instanceId": "57f8e7d45e3540ed9e83cfc0f6636812",
    "createdTime": "2023-08-16T15:32:11.6209051Z",
    "lastUpdatedTime": "2023-08-16T15:32:52.4618299Z",
    "input": null,
    "output": [
        "Hello Oslo!",
        "Hello Reykjavik!"
    ],
    "runtimeStatus": 1,
    "customStatus": null,
    "history": [
        {
            "EventType": "ExecutionStarted",
            "Input": null,
            "Correlation": null,
            "ScheduledStartTime": null,
            "Generation": null,
            "Timestamp": "2023-08-16T15:32:11.6209051Z",
            "FunctionName": "DurableFunctionsOrchestrationCSharp1"
        },
        {
            "EventType": "SubOrchestrationInstanceCompleted",
            "Result": [
                "Hello Tokyo Subway!",
                "Hello London Subway!"
            ],
            "Timestamp": "2023-08-16T15:32:32.2081703Z",
            "ScheduledTime": "2023-08-16T17:32:11.810787+02:00",
            "FunctionName": "SubOrchestrationCSharp1",
            "Input": null
        },
        {
            "EventType": "TaskCompleted",
            "Result": "Hello Oslo!",
            "Timestamp": "2023-08-16T15:32:42.3814044Z",
            "ScheduledTime": "2023-08-16T17:32:32.3416229+02:00",
            "FunctionName": "SayHello",
            "Input": null
        },
        {
            "EventType": "TaskCompleted",
            "Result": "Hello Reykjavik!",
            "Timestamp": "2023-08-16T15:32:52.4449048Z",
            "ScheduledTime": "2023-08-16T17:32:42.424239+02:00",
            "FunctionName": "SayHello",
            "Input": null
        },
        {
            "EventType": "ExecutionCompleted",
            "OrchestrationStatus": "Completed",
            "Result": [
                "Hello Oslo!",
                "Hello Reykjavik!"
            ],
            "FailureDetails": null,
            "Timestamp": "2023-08-16T15:32:52.4614817Z"
        }
    ]
}

You see, no child instanceId (which is supposed to be ea459c8948d74e6fb86635a48c2517a1:0) returned.
What are we missing?

[sebastianburckhardt]

What seems to happen is that IDurableOrchestrationClient.GetStatusAsync is not returning the original history, but a processed version that removes some events and does some event correlation.

I was not aware that this is happening (the code that does this processing is not part of Netherite, it is in the DF extension).

Here is the code that does it:
https://github.com/Azure/azure-functions-durable-extension/blob/2455636cca50cfb4d1543633d28657437bbb5173/src/WebJobs.Extensions.DurableTask/ContextImplementations/DurableClient.cs#L850

Not sure why it does that. Seems unnecessarily complicated.

[StephenAstrumU]

@sebastianburckhardt does it make sense to raise an issue with the DF extension, then, if that's the root cause of this issue?

[sebastianburckhardt]

Yes, I think the appropriate place to modify this would be the code I linked to above.