Unexpected default behavior of QueueClient.OnMessage (ReceiveAndDelete instead of PeekLock) about azure-service-bus-dotnet

[SeanFeldman]

@ncarandini 👍 for better documentation. The current client (not this repo) is doing a horrible job.

But to be fair, this client in this repo is providing the xmldoco, which is an improvement 😃

This is why I had #26 from day one.
/cc @jtaubensee @nemakam

[ncarandini]

Ops, I totally missed it!
I just found the How can we improve the Windows Azure Service Bus? where you explain that the current client is not on GitHub.

Thanks a lot for your clarifying comment.

I'll close this issue as is unrelated to this repo. Sorry for the mess.
I'll also read this library code looking for same behavior and if needed I'll open a new issue, this time with a request for PeekLock default behavior.

[ncarandini]

Hi @jtaubensee, I cloned this repo and saw that the same issue is present here.
Now I've more things to say, pointing to specific code lines of MessageReceiver.cs , OnMessageOptions.cs, MessageReceivePump.cs and QueueClient.cs.
Do you prefer me to open a new issue or reopen this one, removing (by you, if you'll think is ok) the invalid tag ?

[ncarandini]

Thanks @jtaubensee,
I'll procede here :-)

[jtaubensee]

Hi @ncarandini -

So the default behavior is still peek lock, however we default the value OnMessageSettings.AutoComplete to true. This means that when the OnMessage callback exits, Message.Complete is called. If you were to call Message.Abandon or Message.Defer inside of your OnMessage callback, the message would still exist on the queue.

ReceiveDelete means that when the message is retrieved from the queue, it is also deleted from the queue. While this performance can be better, it can result in message loss, if your receiver dies while processing the message.

Does that make sense? To your point, our documentation could do a better job of explaining this.

[ncarandini]

As stated before, I do think that the default behavior of the OnMessage should be PeekLook and this will not be the case if the AutoComplete is set to true.

These are my proposals:

Set AutoComplete to false as the default value

Looking at the OnMessageOptions.cs we see that the constructor set the default options:

public OnMessageOptions()
{
    this.MaxConcurrentCalls = 1;
    this.AutoComplete = true;
    this.ReceiveTimeOut = Constants.DefaultOperationTimeout;
    this.AutoRenewTimeout = Constants.ClientPumpRenewLockTimeout;
}

The easiest things will be to change it to:

...
this.AutoComplete = false;
...

or to add a line in the Microsoft.Azure.ServiceBusCostants class and reference it:

...
this.AutoComplete = Constants.DefaultAutoComplete;
...

namespace Microsoft.Azure.ServiceBus
{
    using System;
    internal static class Constants
    {
        ...
        public const bool DefaultAutoComplete = false;
        ...
    }
}

Inherit the value of AutoComplete from the Mode value of the MessageReceiver (of the QueueClient)

From now on, I'll refer only to the MessageReceiver, because calling the QueueClient.OnMessageAsync() method calls the same method of the QueueClient.InnerReceiver (of type MessageReceiver) and because I take for granted that the QueueClient.InnerReceiver.Mode property will have the same value of the QueueClient.Mode one.

That being said, calling te MessageReceiver.OnMessageAsync() method without the optional OnMessageOptions parameter, will actually produce a call to the more complete method passing a new instance of OnMessageOptions class with the default values created as we've seen before with the notable exception of the ReceiveTimeOut, that is inherited from the value of the corresponding property of the caller (i.e. the MessageReceiver instance):

public void OnMessageAsync(Func<BrokeredMessage, CancellationToken, Task> callback)
{
    this.OnMessageAsync(callback, new OnMessageOptions() {
        ReceiveTimeOut = this.OperationTimeout
    });
}

My suggestion is to inherit also the AutoComplete from the caller Mode value, with a very simple type conversion:

public void OnMessageAsync(Func<BrokeredMessage, CancellationToken, Task> callback)
{
    this.OnMessageAsync(callback, new OnMessageOptions() {
        ReceiveTimeOut = this.OperationTimeout,
        AutoComplete = this.ReceiveMode == ReceiveMode.ReceiveAndDelete
    });
}

@jtaubensee, if you think this has some meaning, I can make a PR of it.

[jtaubensee]

@ncarandini - Did you see my previous message? I understand what you are saying, however there seems to be a disconnect.

Today, there are two different receive modes in Service Bus. Below is a description of each:

• PeekLock:

This operation atomically retrieves and locks a message from a queue or subscription for processing. The message is guaranteed not to be delivered to other receivers (on the same queue or subscription only) during the lock duration specified in the queue/subscription description. When the lock expires, the message becomes available to other receivers. In order to complete processing of the message, the receiver should issue a delete command with the lock ID received from this operation. To abandon processing of the message and unlock it for other receivers, an Unlock Message command should be issued, otherwise the lock duration period can expire.

This operation should be used in applications that require At-Least-Once delivery assurances. If the receiver does not delete the message before processing succeeds, this ensures that another receiver is able to attempt processing after the lock duration period expires.

• ReceiveAndDelete:

This operation receives a message from a queue or subscription, and removes the message from that queue or subscription in one atomic operation.

With that said, I believe this is different from what you are talking about. Aren't you basically suggesting that OnMessageOptions.AutoComplete should be set to false by default? Here is a description of AutoComplete.

The reason that it is set to true, is so that you don't always have to call Message.Complete inside of OnMessage. In the past we had some customers who forgot to do so, and then a message would keep appearing, until it was eventually dead-lettered. We really are promoting OnMessage as the easiest way to get up and running with Service Bus. It can be hard to develop your own receive pump, and we see customers who do it incorrectly all the time. They can very easily get a high bill because they call too often or inefficiently.

The only place that AutoComplete is used is here:

if (this.messageReceiver.ReceiveMode == ReceiveMode.PeekLock &&
	this.registerHandlerOptions.AutoComplete)
{
	await this.messageReceiver.CompleteAsync(new[] { message.LockToken }).ConfigureAwait(false);
}

which in turn is then called here, after OnMessage has completed.

[ncarandini]

Hi @jtaubensee,
thanks for your kind explanation, now I can better understand the historical reasons of the default value of true of the AutoComplete option.

Nevertheless, while I can understand this is a safety rail for beginners, I still think this is misleading for most developers. On my opinion, using the library message pump vs. manually requesting a message shouldn't change the behaviour set by the ReceiveMode, unless I (as a developer) opt to the safety rail of AutoComplete = true.

Anyway, on the current .NET library (I still don't know if also in the current repro, I haven't tested it yet), there is no difference from having (ReceiveMode = PeekLook & AutoComplete = true) versus (ReceiveMode = ReceiveAndDelete) because the message is removed from the queue even if the handler method got an ErrorException.

Here's my test code:

...
client.OnMessage(receivedMessage =>
{
    try
    {
        // To check the OnMessage behaviour we force an Exception Error
        throw new Exception("Exception thrown to test message pump behavior");

        // Do something with the message

        // We are using the message pump that has the AutoReceive set to true by default.
        // No need to call receivedMessage.Complete();
    }
    catch (Exception ex)
    {
        WriteLineColor($"Error processing MessageId {receivedMessage.MessageId}: {ex.Message}", ConsoleColor.Red);
        Console.WriteLine();
    }
});
...

If confirmed, this must be a bug.

[jtaubensee]

@ncarandini - To be clear, AutoComplete is only ever used for OnMessage / Receive pump (which are two words for the same thing). AutoComplete is not a property for ReceiveAsync().

What we are talking about here are two different things. Just because AutoComplete is set to true, does not mean that the ReceiveMode becomes ReceiveDelete. It simply means that we call Message.Complete() for you when your lambda is done executing. If there is an error inside of that lamda, we do not call complete, and the message is available for processing again on the queue.

If you have ReceiveMode set ReceiveDelete then there is never a reason to call Message.Complete as the message is already removed from the queue.

Can you share your code that shows AutoComplete when there is an exception in your OnMessageHandler? FYI - the implementations of that message pump / OnMessage are the same between this library and the .NET Framework NuGet package.

[ncarandini]

Yes, AutoComplete is indeed a property of OnMessageOptions and it solely purpose is to call message.Complete inside the OnMessage code, only if it is true AND the handled method works fine (i.e. no ErrorException thrown). This is (finally) clear to me.

My implementation was wrong, because I was catching the error so the handled method completed successfully.

So, at the end of the story, only better documentation is needed to explain why, when and how to set AutoComplete to false.

Even the ReceiveSample use it:

// Register a OnMessage callback
queueClient.RegisterMessageHandler(
    async (message, token) =>
    {
        // Process the message
        Console.WriteLine($"Received message: SequenceNumber:{message.SequenceNumber} Body:{message.GetBody<string>()}");

        // Complete the message so that it is not received again.
        // This can be done only if the queueClient is opened in ReceiveMode.PeekLock mode.
        await queueClient.CompleteAsync(message.LockToken);
    },
    new RegisterHandlerOptions() {MaxConcurrentCalls = 1, AutoComplete = false});

Thanks for all the time spent on answering my question.

[SeanFeldman]

The swallowed exception has tricked you 😃

[ncarandini]

I must wear the donkey hat for at least a week!