Messages ordering when receiving batch of messages from Service Bus about azure-sdk-for-net

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jsquire]

Hi @junaidahmed92. Thanks for reaching out and we regret that you're experiencing difficulties. The client will always return messages to you in the order that the Service Bus returns them. What you need to be aware of, however, is that the Service Bus service itself does not guarantee message ordering unless sessions are used.

Assuming no errors and normal operation, you'll usually see messages streamed to the client in the order that they were received by the service. However, messages which are locked by a reader are not sent by the service and, thus, may appear out-of-order if the holder of the lock does not complete the message. For example, Reader A and Reader B are both reading from a queue. Only one reader has control of a message at any time. Reader A will not see the messages locked by Reader B and vice-versa. However, if Reader B abandons a message that it has read, that message stays in the queue any may be read by Reader A who would see it out-of-order.

This is also true if your application has the message, but the lock expires before you complete it. For example, Reader A gets messages 1, 2, 3, 4 from the service. Processing takes a while and the message lock expires before message 4 is done processing. When Reader A goes to complete message 4, the service will not allow it. The next time Reader A reads, it will see message 4 again, out-of-order.

As for choosing a session identifier, anything that is unique and can be used to group related messages together would work. More information can be found in First-in, first-out (FIFO) pattern.

[github-actions]

Hi @junaidahmed92. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.

[junaidahmed92]

Thanks @jsquire for tbe detailed answer.
So this means if i add a same sessionid to all messages that are picked by a specific topic subscription, i will receive them in order.
E.g I've a Device-subscription and I add a sessionid "device-session" to all the messages.

[jsquire]

Thanks @jsquire for tbe detailed answer. So this means if i add a same sessionid to all messages that are picked by a specific topic subscription, i will receive them in order. E.g I've a Device-subscription and I add a sessionid "device-session" to all the messages.

Correct, assuming that you're not receiving/processing in parallel and prefetch is not enabled.

[junaidahmed92]

No, I'm not processing in parallel. I am fetching a batch of messages and processing them in loop.
However prefetch is enabled on the ServiceBusSessionReceiver. Should I disable it?

[jsquire]

However prefetch is enabled on the ServiceBusSessionReceiver Should I disable it?

If strict ordering guarantees are needed, yes. There's a known issue in the AMQP transport library that we use which causes prefetch + sessions to ignore ordering in a rare edge case. The team that owns the transport is investigating, but the behavior has only been observed in one application and is non-deterministic so they're still attempting to find the root cause.

[junaidahmed92]

Thanks @jsquire for the clarification. I've changed my implementation. However I've got a question, what if I've only receiver who is responsible to receive all messages and if there's an error or a lockdown expires, the same receiver will receive and process it again. Are using sessions still preferable in this scenario?

[junaidahmed92]

I've got couple of more questions:

1. I've set a peek lock time for 5 mins and my SessionReceiver's receivemode is set to PeekLock but even after a min I can fetch the same message again. This could raise a problem if I've multiple listeners and one is already processing a message.

ServiceBusSessionReceiver receiver = await _serviceBusClient.AcceptNextSessionAsync(topicName, subscriptionName, new ServiceBusSessionReceiverOptions
{
    ReceiveMode = ServiceBusReceiveMode.PeekLock
});

2. The other question is related to the delivery count. I have noticed that even if the message processing throws exception, the delivery count always remains 0 and based on the this doc, it's expected. But the problem is, if a single message in a session is unable to get processed, it will keep on retrying forever and will never goes to the DLQ. What are the options to tackle this? I've set a DeliveryCount to 5 and I want to throw the session message to DLQ after 5 attempts. But if on retry the DeliveryCount is always 1 if I attach a breakpoint.

[jsquire]

1. If the session lock is held, you cannot read a message for the same session with two receivers. If you abandon the message but still hold the session lock, that message is available to be read immediately but only by the receiver holding the session lock. There are no individual message locks when using sessions - the session itself is the lock.

2. Delivery count is a broker-owned concept that the client library has no insight into nor influence over. The Azure SDK team cannot speak to the behavior here authoritatively. One thing that I can mention is that you do have the ability to manually dead-letter a message and do not need to rely on the broker to do so via delivery count semantics.

[junaidahmed92]

Thanks for the clarification. But when I wss not using sessions the delivery count was completely fine. With ServiceBusReader i mean.
I didn't have to manually retry and DLQ the messages but rather service bus was doing that.

Also can you confirm if I only have a single ServiceBusReceiver, is tbe messaging ordering still a problem without sessions.

[jsquire]

Thanks for the clarification. But when I wss not using sessions the delivery count was completely fine. With ServiceBusReader i mean.
I didn't have to manually retry and DLQ the messages but rather service bus was doing that.

Delivery count is a broker-owned field and actions taken are by the service; the Azure SDK clients don't play a part in how it works - sessions or no. It's all service behavior, so there's nothing that I can offer beyond what I've already shared.

Also can you confirm if I only have a single ServiceBusReceiver, is tbe messaging ordering still a problem without sessions.

Yes, for the reasons that I mentioned above. Any scenario where the lock expires, you have an exception in your processing, your application or its host crashes/reboots/migrates can cause messages to be delivered out of order. They're edge cases, but not uncommon ones. The salient point is that the Service Bus service makes no guarantees about strict ordering unless sessions are used.

Here, again, the client has no insight nor influence; we just return what the service gave us in the order that it gave it to us.

[junaidahmed92]

Alright, thanks. Closing this issue now.