Question: Event Hub traffic when no orchestrations are executing about durabletask-netherite

[sebastianburckhardt]

Some continuous background traffic is expected while there are active hosts. In idle state, a partition sends a message every 10 seconds to the load monitor.

Is this a consumption plan? On a consumption plan, the task hub scales to zero when everything is idle so there should not be any active hosts and you should not see any messages.

Also, the numbers you show in the picture above seem a bit high, I would expect less than 10k messages a day given that there is just one partition. This may indicate that there is some problem.

Sometimes there are issues with requests that get stuck. What does the partition table show? If your taskhub is idle, the columns Instances, WorkItems, Timers, Requests and Outbox should be all zero.

[nls-jbarntish]

In our case, the DF stack needs to be on a VNET so we cant use consumption. Its currently on an EP2 that scales out to a max of 2 nodes.

Regarding the partition table: there are 12 records. All columns are zero except Instances, all of which in a range between 5100 and 5350.

[jbarntish]

Whoops, accidentally switched user contexts in my previous post ...☝️

Also worth noting: the load on event hub is still present. Would a taskhub reset clear this out?

[jbarntish]

After resetting the task hub, the event hub traffic continued until stopping the DF function app. Once stopped:

• The task hub files and folder was recreated (as indicated in the doc)
• DurableTaskPartitions table refreshed and Instances were set to zero
• Event hub traffic dropped to nothing

After letting it sit for a short period and starting it again, it spun back up again to the same level. No orchestrations were started manually or via schedule (such as a cron trigger). All of the fields in the table you mentioned are still at zero.

I would expect less than 10k messages a day given that there is just one partition. This may indicate that there is some problem.

Our app is configured with 12 partitions as per the recommendation here. Your statement about 10k/partition makes me think we might be seeing ~120k events a day due to the 12 partitions. Is that accurate?

Also worth mentioning: our EP2 is setup to scale out to 4 nodes (not 2 as previously mentioned). Should the # of partitions typically match the node scale out setting?

[sebastianburckhardt]

Your statement about 10k/partition makes me think we might be seeing ~120k events a day due to the 12 partitions. Is that accurate?

It is about 8.6k messages per partition per day so about 103k messages for 12 partitions a day.

Should the # of partitions typically match the node scale out setting?

number of partitions should be set to the MAX scaleout you want to handle, so generally larger than just the number of nodes you are running at the moment.