Concurrency exception during back end processing about cqrslite

[petersondrew]

To me, this is more appropriately handled in ServiceA and ServiceB. If your command is rejected due to a concurrency exception, they should be the ones reloading the read model and making the decision to resubmit the command. It may even be possible that they submit the command differently due to other changes that were made.

[alex-wheat]

Yes, handling these issues in ServiceA and ServiceB is possible solution, and I was thinking about it already, but I'm expecting more than couple services like these and handling the issues in every of them probably will be a bit more difficult than just having some universal mechanism that will take care about it.

Cause the processing goes on back end without human involved, the order of events it's not really important - consistency and versioning is important, but order of events that I operate with is not really important. So, having some universal mechanism for back end processing would be very beneficial for me.

[alex-wheat]

Also, it doesn't look like possible to use already existing default Session from CQRSlite framework even if I decide to handle the issues on services' side. Let's take a look on the code...

here is my initial code

public void Handle(StatusChanged message)
{
    var file = session.Get<DepositionFile>(message.FileId);

    switch (message.Status)
    {
        case RecordStatus.Processed:
            file.RecordProcessed(message.Id);
            session.Commit();
            break;

        case RecordStatus.Failed:
            file.RecordFailed(message.Id);
            session.Commit();
            break;

        default:
            break;
    }
}

now I try to wrap every Commit with try/catch like this

public void Handle(StatusChanged message)
{
    var file = session.Get<DepositionFile>(message.FileId);

    switch (message.Status)
    {
        case RecordStatus.Processed:
            file.RecordProcessed(message.Id);

            try
            {
                session.Commit();
            }
            catch (Exception)
            {
                //  deposition file instance still the same...
                file = session.Get<DepositionFile>(message.FileId);
            }
            break;

        case RecordStatus.Failed:
            file.RecordFailed(message.Id);
            session.Commit();
            break;

        default:
            break;
    }
}

and I can't load fresh deposition file instance cause it wasn't removed from session and there is no any mechanism to refresh it...

That means that I have to implement my own Session anyway and in this case it's not clear why not to implement it with kinda universal mechanism and try to resubmit all changes :)

[petersondrew]

You shouldn't try to handle it in your Aggregate, this should be handled in client code. Here's a quick and dirty example with retry on concurrency exception.

do
{
    try
    {
        var data = _readModel.Fetch<SomeData>();
        var command = new ChangeStatusCommand(RecordStatus.Processed, data.version);
        _commandSender.Send(command);
        break;
    }
    catch (ConcurrencyException)
    {
         // ignore, retrying
    }
    catch (Exception)
    {
         throw;
    }
} while (true);

[alex-wheat]

sorry for the delayed response... now I'm a bit confused. Why I shouldn't try to handle it in Aggregate?

Let me share a bit more info about my workflow...
I'm trying to process large files that contains some logical records inside. I have two Aggregates: DepositionFile and Record. When I split file on Records I periodically pushing parsed records to event store that issue events to bus that new records are ready for processing.
Record’s processing starts as soon as RecordCreated event appeared in a bus. Result of record’s processing either Success or Fail. In case of Success, RecordProcessed event will be issued to the bus and in other case - RecordFailed.
Now I need to figure out when the processing is finished. In order to do this on every RecordProcessed or RecordFaild I update DepositionFile aggregate and increment appropriate counter. When the sum of failed records and processed records equals to total records, I assume that the processing is finished. And all this business logic is inside DepositionFile aggregate.

The questions that I currently have:

1. Why I shouldn’t handle it in Aggregate?
2. What is bad in my workflow?

[petersondrew]

I'm having a little trouble understanding your architecture, but something seems off. If you receive a CreateRecord command, and publish a RecordCreated event, as long as you use the expectedVersion property in the client code then you do not have to deal with concurrency exceptions in your domain logic (and I would argue, should not deal with it there for better separation of concerns).

I have a few questions regarding these statements:

When I split file on Records I periodically pushing parsed records to event store that issue events to bus that new records are ready for processing

record’s processing starts as soon as RecordCreated event appeared in a bus. Result of record’s processing either Success or Fail. In case of Success, RecordProcessed event will be issued to the bus and in other case - RecordFailed.

Are you using Sagas to handle this process? If not, that is definitely part of your problem. Commands should prompt aggregates to publish events, which can be picked up by Sagas, which themselves can generate commands for other aggregates. Nowhere in the process should something generate an event in direct response to an event, there should always be a command published.

I think I understand your ultimate goal, which is for client code to know when all records have been processed because it has no visibility into the processing of the records (again, it sounds like there should be a Saga here, but you haven't mentioned using one so I'm unsure if you are already). To achieve that goal, I think what you should be doing is updating a count on a read model, rather than on the aggregate. A simple read model containing something akin to a dictionary of {DepositionId, ProcessedCount}. When the read model receives the RecordProcessed and RecordFailed events for that DepositionId it can update the ProcessedCount. If you follow the advice of handling concurrency errors on the command publishing side, and ensure you're using an aggregate, you will never miss counts in your read model due to concurrency issues. From that point I'll leave it up to you to determine how the client polls the read model to know when the ProcessedCount reaches the expected count.

Let me know if I've misunderstood something about the concurrency issue you're having or the domain that you're working with.

[alex-wheat]

Drew, thanks for the detailed response and paying your attention to my problem! Appreciate it!

Just a some quick answers before I'll go through your suggestions... :)

First of all, I'm not using Sagas. And I did it on purpose. I used to play a bit with Sagas from MassTransit, (and initially I tried the same processing with a MT), but had a problem to use it together with CQRSlite. I wasn't able to implement Bus using MT. So, I gave up that idea and now using RabbitMQ that I didn't have any problems with implementation. Also, Sagas from MT looks a bit complicated. And still doesn't have .NET Core implementation.

Also I have a feeling that I have very simple processing pipeline that using Sagas is not really needed,
but maybe I'm wrong... :)

Do you have any experience with using CQRSlite with MT together? If you could share your experience it would be very helpful! :)

[petersondrew]

I have not tried to use CQRSlite with MT - but you definitely don't need the full weight of MT to implement Sagas. I think part of the confusion may be the overloaded use of the term Saga outside of CQRS. In CQRS, it is also sometimes referred to as a Process Manager (I'll refer to it as such from here on out to avoid confusion).

You could actually implement your own process manager using CQRSlite fairly easily. What you need is an EventHandler that publishes new commands on the bus, that's it! That doesn't handle things like repeat events, etc, but may be enough for your case. (See articles below if you need to store state in your process managers for complex workflows).

Microsoft actually has a nice series on CQRS patterns, here is one on Process Managers. There are also these two articles by Jonathan Oliver CQRS Sagas with Event Sourcing I CQRS Sagas with Event Sourcing II which detail a more involved Process Manager that behaves more like an Aggregate. I would read those and decide where your problem lies on the spectrum.