Extending to allow tracking of finished handlers and sagas about cqrs

[raymondsze]

@boenrobot
Thanks for the long reply.

I know nestjs/cqrs won't provide the persistent storage mechanism and this should left to developer decision.

What I am seeing, is there enough extension point we could do the persistent mechanism that fit our need?

For each event handler, we should have ability to control how events sent to the current in-memeory eventBus provided by this library. Yes, we have the event publisher for this purpose although I think the override way that nestjs/cqrs providing is not a nice way. What I expect is provide the publisher via some configuration like dynamic module, or use some decorator to provide the event publisher.

The same story should apply to commandBus (commandHandler) and queryBus (queryHandler) as well.

Ok, now lets talk about Saga. The current implementation subscribe the whole event stream. First, I'm not sure by definition that saga should take care of two or more prior events. What I hear more is using state machine to implement the saga, when one event comes, there must be an update of state that could wait for the next event, whether we should concern about the ordering of events depends on how developer decise for the state management.

The idempotency identifier of commands based on sequence order is just one example I think of. Of course, it is also possible to derive from the event itself, I think the idea behind is the same.

The only difference is I prefer treat saga as same as an aggregate root that use event for the update instead of direct modification. Anyway this is still possible for developer to extend from current implementation (using event handler instead) of nestjs/cqrs.

The overall problems I see in this library:

1. Unable to override commandBus and queryBus.
2. Should have more "nice" way to override eventPublisher. (Actually I'm not sure why not allow developers override the whole eventBus but separate a eventPublisher component)
3. Saga should not directly use the commandBus. There should be an extension point allow developer to modify the way how saga publish the command to commandBus. (This may be solved if problem 1 is solved)
4. Need a way to get the command exception reply. For this, I think if command is issued by saga, the command handler should reply the exception as event so saga could subscribe and handle it back. So we need to have a way to know whether the command is dispatched asynchronously or synchronously, this seems could be solved if both problem 1 and 3 are solved.
5. The ack issue, I have no idea here. But for all event handler and saga, we should have a extension point to know that all the event related handlers are handled so we could send the ack to the message broker. This is possible for Event Handler if your PR is merged but not Saga.

And, by discussion, seems like I would never use the @saga but use multiple event handlers instead. The saga provided by this library is quite different from what I see in other framework except the naming is same. I see no point to use the @saga from this moment, it just make everything harder and somehow the naming forcing me to implement the Saga pattern using @saga.

[marcus-sa]

@boenrobot NestJS sagas doesn't implement the orchestration pattern, but rather closer to the choreography pattern.

There's several ways of using Convoy:

• Transactional outbox pattern.
• Message brokers (Kafka is implemented for both publishing and subscribing)
• Event Sourcing (not implemented yet)
• In Memory (should only be used for testing)

All of these implementations use MikroORM, to ensure that messages are transactional & idempotent, by not processing duplicate messages.

The goal is a 1:1 port of Eventuate Tram + Sagas with high availability for NestJS, which indeed solves your issue in the correct manner.

There's a lot of stuff in Eventuate Tram that isn't documented either, so this'll also take a long time to do.

[Eraledm]

Any news?

[raymondsze]

Inspired by what Axon/Akka did. How about if we could implement like this way... I'm not sure it is possible or not.

@Command("Order")
class CreateOrder implements ICommand {
  @TargetAggregateIdentifer()
  aggregateId = () => payload.id;

  constructor(private readonly payload: CreateOrderPayload) {}
}

@Event("Order")
class OrderCreated implements IEvent {
  @TargetAggregateIdentifer()
  aggregateId = () => payload.id;

  constructor(private readonly payload: OrderCreatedPayload) {}
}

@AggregateState('Order')
class OrderState {
   @AggregateIdentifier()
   public readonly id: string;

   public status: 'PENDING' | 'APPRIVED';

   @EventSourcingHandler(OrderCreated)
   orderCreated(@Evt() event: OrderCreated) {
      this.status = "PENDING";
   }

   @EventSourcingHandler(OrderApproved)
   orderApproved(@Evt() event: OrderApproved) {
      this.status = "APPROVED";
   }
}

@AggregateRoot("Order")
@AggregateState(OrderState)
class Order extends AggregateRoot {
   @CommandHandler(CreateOrder)
   createOrder(@Cmd() command: CreateOrder, @State() state: OrderState) {
      return [new OrderCreated(...)];
   }

   @CommandHandler(ApproveOrder)
   approveOrder(@Cmd() command: CreateOrder, @State() state: OrderState) {
      return [new OrderApproved(...)];
   }
}

@AggregateState('CreateOrder')
class CreateOrderState {
   @AggregateIdentifier()
   public readonly id: string;

   public status: 'STARTED' | 'ENDED';

   @EventSourcingHandler(OrderCreated)
   orderCreated(@Evt() event: OrderCreated) {
      this.status = "STARTED";
   }

   @EventSourcingHandler(OrderApproved)
   orderCreated(@Evt() event: OrderApproved) {
      this.status = "ENDED";
   }
}

@AggregateRoot("CreateOrder")
@AggregateState(CreateOrderState)
class CreateOrderSaga extends AggregateRoot {
  @SagaEventHandler(OrderCreated)
  handleOrderCreated(@Evt() event: OrderCreated, @State() state: CreateOrderState) {
      return new ApproveOrder(...);
  }
}

class OrderEventsSubscriber {
    @EventSubscriber(OrderCreated)
    handleOrderCreated(@Evt() event: OrderCreated) {}

    @EventSubscriber(OrderApproved)
    handleOrderCreated(@Evt() event: OrderCreated) {}
}

[marcus-sa]

@boenrobot I actually went ahead and created a port of https://github.com/eventuate-tram/eventuate-tram-sagas for NestJS which you can find here https://github.com/marcus-sa/nest-convoy/tree/dev
So far the https://github.com/marcus-sa/nest-convoy/tree/dev/examples/sagas-customers-orders works as expected, but there's still a lot of stuff to be done, like documentation & unit testing, support for transactional outbox & event sourcing etc.

I didn't really see the need to reimplement the wheel, when Eventuate already provides the strongest fundament.

[boenrobot]

@marcus-sa I don't know about your implementation, but eventuate's model seems to serve a different use case than NestJS' sagas. NestJS sagas orchestrate events from NestJS itself, and ultimately triggers commands from the app itself. Eventuate sagas on the other hand seem to be all about connecting multiple (microservice) apps.

That said, if your implementation can be hooked to a custom storage engine (e.g. an SQL DB) so that it persists across NodeJS restarts, and with minimal effort, I'd happily use it, even if there's an extra runtime overhead due to the extra saga participants orchestration (assuming a 1:1 port of eventuate sagas).

[justinpenguin45]

Any updates on this? It would be great to have someone from Nestjs weigh in on this. How do we coordinate a more complex scenario, f.e. more than 1 event type in the saga. Transactional behavior?

[kamilmysliwiec]

I would like to create a custom event storage mechanism, whereby I track when event handlers were dispatched, when each event handler finished executing, and when all registered sagas related to an event have finished executing.

This package offers a reactive architecture based on RxJS. It doesn't provide any sophisticated solution in terms of tracking what and when has been executed. We also don't plan to introduce more complex features - requirements vary depending on the application and it's up to the development team to add more tailored features that are required in a specific project.

I would be willing to write such an extension guide myself (as an extra recipe?) after I actually make one that can do this sort of tracking.

PRs are always welcome - to both documentation (recipe section) as well as this package specifically.

On a related note, I also have a question that the docs don't make clear... Do sagas even wait for the handlers of an event to finish executing, or is the mere act of dispatching an event enough to trigger all sagas that track it? From reading the sources, it seems like the answer to that is "No, sagas don't wait for event handlers to finish. Sagas are executed as soon as all tracked events are dispatched".

It depends on the Scheduler, see:

• https://blog.strongbrew.io/what-are-schedulers-in-rxjs/
• this also might be helpful https://rxjs-dev.firebaseapp.com/api/operators/observeOn.

[boenrobot]

It doesn't provide any sophisticated solution in terms of tracking what and when has been executed. We also don't plan to introduce more complex features - requirements vary depending on the application and it's up to the development team to add more tailored features that are required in a specific project.

I understand that, and I'm not expecting it to ever have that built in.

I am however hoping (and this is the feature request here) that there will be enough extension points for that to be doable by setters for custom classes at module initialization (as with the publisher), or at the very least, classes that extend the ones in this package and export them into a custom module.

It depends on the Scheduler

So saga execution should be tracked separately and independently from handler execution. Got it. Thanks.

[raymondsze]

Hi @boenrobot, @kamilmysliwiec,
I am trying to do cqrs using Kafka as the eventBus. I also need a way to know whether all the event handlers and the saga corresponding to the event are executed.

I have questions on that, using explorer service, I could know how many event handlers have handled the event so I could commit the offset (like ack in rabbitMQ) until all the event handlers have be successfully executed. But this is not possible for saga, the saga handler only receive the event stream and filter the event using "ofType". The explorer service have no knowledge on what event the saga is subscribing to...

Let's say all the event-handlers have been successfully executed, offset is committed. At the moment, the saga run into failure because of unexpected reason. After a restart, the saga can never pick up the event to continue the process....

Is there any reason why the saga handler designed like this but not just same as event handler? Even it is possible to change the event publisher used by event-bus, it is not possible to make the event-handlers and saga execution transactional.

And, is it reasonable I inject the commandBus inside the event-handler to perform what sagas do?

[boenrobot]

@raymondsze I'm aware of this, and have wondered myself how best to solve it (which is a large part of why the PR about this is still WIP). Part of why it's difficult is that I didn't even know how to monitor multiple events as is (see nestjs/docs.nestjs.com#903).

At this point, I'm thinking it might be best if each saga also gets a new class (let's say called a SagaManager) that can save, load and clear named groups of events.

This would allow the event store to group events into the store rather than letting grouped observables do it, which is more scalable if your grouping criteria is a user, rather than a feature.

So within a saga, you would need to filter the events to be saved (like you do now), then call the manager's save method with a grouping key. Then, return an observable from the manager that would emit when a scan criteria over a group is complete (and would scan on each saved event), keeping in mind that within each scan, the sequence of the group itself would be subject to changes. Then pipe a map to that sequence that would return the commands to be executed.

At this point, the fact that multiple commands can also be executed is sort of tripping me up though. We want to clear the events from the store only after all commands have finished executing, but currently, each individual command is executed without it being aware of what was the last emitted event, or even waiting for completion.

Maybe we could make command execution part of the manager too, then just return an empty observable for the current one's sake, except that last part sounds like unnecessary extra piece of code to be maintained, just to keep BC.

[raymondsze]

@boenrobot
I'm still studying cqrs and doing a poc example on it. I found lots of framework in Java and .Net world use the dependency injection pattern to implement their framework. So I think nestjs/cqrs could be a good fit to do this.

Example references:
Java Axon: http://progressivecoder.com/saga-pattern-implementation-axon-spring-boot-part-2/
.Net Akka: https://akkatecture.net/docs/your-first-aggregate

Here is what I plan to do and I wanna ask some opinions.

Instead of using Saga provide by nestjs/cqrs, I use EventHandler instead. For Sagas, there is orchestrator pattern and choreography pattern. Let's ignore choreography first as I have read many books that do not recommend it if we could do the orchestrator pattern.
Reference: https://microservices.io/patterns/data/saga.html

To do orchestrator pattern, I treat Sagas as a special kind of aggregateRoot (since it has state to determine what command it will send for next step), Akka in .NET also call Saga as "AggregateSaga".

First, create a aggregateRoot called CreateOrderAggregateSaga and also its repository CreateOrderRespository that is responsible to query the AggregateSaga by id from the event-store (i.e loadFromHistory in aggregateRoot).

Create a command called "StartCreateOrder" and the corresponding commandHandler. The commandHandler inject the CreateOrderRespository to query the CreateOrderAggregateSaga, and apply event which called "CreateOrderStarted" (the CreateOrderAggregateSaga need to implement onCreateOrderStarted the change the saga state), then commit to eventBus.

The EventHandler of CreateOrderStarted inject the commandBus and execute CreateOrder command (need include to metadata telling how to reply this Saga) which belongs to order service. The CommandHandler of CreateOrder would process the command and emit OrderCreated (PS: the metadata have to be propagated in the whole process).

The EventHandler of OrderCreated inject the commandBus and check if metadata exists in the OrderCreated event. If metadata exists, call commandBus to execute a UpdateCreateOrderSaga command which contain the event payload "OrderCreated".

The CommandHandler of UpdateCreateOrderSaga would emit a CreateOrderSagaUpdated event (the CreateOrderAggregateSaga need to implement onCreateOrderSagaUpdated the change the saga state). The event handler of CreateOrderSagaUpdated inject the commandBus and send the next command according to the aggregate saga state. Lets say, the commandBus execute a ApproveOrder command which belongs to order service.

Afterward, its same as what we do in CreateOrder command recently, and do util the saga have no command to send, it means the saga is completed.

The metadata format reference: https://blog.arkency.com/correlation-id-and-causation-id-in-evented-systems/

And then, all actions in cqrs are very concern about "atomic". There is a problem in nestjs/cqrs. The aggregateRoot commit method loop over the uncommitted events and send them separately. It does not make use of the "publishAll" provided in eventPublisher. I ended up using eventPublisher.publishAll(aggregate.getUnCommitedEvents()) instead of aggregate.commit().

I'm not sure saga should be implemented in this way or not. And one more thing, the cqrs pattern normally used in micro-service architecture. Usually, services are designed as per aggregateRoot. If I treat saga as special kind of aggregate, it means the CreateOrderSaga and the Order are not belongs to the same service which is possible in micro-service world. The problem is the commandBus now using in nestjs/cqrs is a synchronised in-memory bus. Logically, the command validation logic should belong to the aggregate service itself (i.e the saga service should know nothing about the order service, but just send the command to order service to its entry point).

As we cannot switch the commandBus easily now. What I think the possible solution is make use of transport, in the subscriber side, just do commandBus.execute. Then we change all the "commanBus.execute" mentioned before to "client.send(command)". Same situation happened in queryBus as well.

Thanks a lot.

[boenrobot]

The above looks like way too much boilerplate for what is just one piece of the puzzle - being able to keep a sequence of events across restarts, until the saga is ready to handle them, and only archiving them once it's handled by all sagas that may need it.

The above seems like you're ditching the whole CQRS module altogether, in favor of a completely different sort of library, just because of this (admittedly important) missing aspect of it.

I think a new class with a decorator might be sufficient to address that particular case, though that does mean that we have saga properties and saga classes, working in a different way.

I'm thinking something like the following:

@Saga('MySaga'/*Optional name in storage, used for grouping prefix. Defaults to name of class.*/)
class MySaga<EventType extends IEvent> implmenents TrackableSaga<EventType> {
  /**
   * Support for Nest DI
   */
  public constructor(praivate readonly databaseService: DatabaseService) {}

  /**
   * Similar to the first part in a saga property.
   * Tracker may or may not preserve events as early as each emit from this step.
   * That is tracker dependent.
   */
  public filter(stream: Observable<EventType>): Observable<EventType> {
    return stream.pipe(ofType(MyEvent, MyOtherEvent), filter((e: MyEvent | MyOtherEvent) => true/* other filters */));
  }

  /**
   * Optional method. Determines the group the event belongs to.
   *
   * Unlike using a groupBy rxjs operator, this one does not create a new observable
   * in memory per group.
   * Instead, the tracker must add the event to a persistent group in storage,
   * and create the group if it doesn't exist already.
   * The group names are namespaced by the saga name.
   *
   * Some trackers may choose to only preserve the event after this point if it exists,
   * while others may preserve the event earlier, and just add an association here.
   *
   * As far as nestjs/cqrs is concerned, it should trigger this method for each emit of filter,
   * if it exists, or skip to scanBy with groupKey being undefined,
   * effectively treating the entire sequence as belonging to a single group.
   */
  public groupBy(e: EventType): string {
    return e.userId; // The fact that there's no observable in memory enables grouping like this to scale.
  }

  /**
   * Optional method.
   *
   * Every time an event is added to a group, this method is called with
   * the contents of the entire group, in the order they exist in the group at the time.
   *
   * This method should return the new contents of the group.
   * It may clear the entire sequence, reorder events, or even add new events
   * from other locations.
   *
   * If this method is present, after its execution, trackers should archive any
   * events that were present in the input, but not in the output,
   * preserve any events that are in the output but not in the input,
   * and reorder the sequence in accordance with the output.
   *
   * On startup, this method should be called for each group still in storage.
   * If not present, execution can skip to isComplete.
   */
  public scanBy(sequence: Readonly<EventType[]>, groupKey?: string): Readonly<EventType[]> {
    return sequence;
  }

  /**
   * Takes a sequence of events as they appear after a scan (if there is one)
   * and determine if they are ready to be given to commands.
   *
   * If this method returns false, the sequence may be unloaded
   * from memory until the next event in the group appears,
   * at which point they should be reloaded from the storage,
   * and go through scanBy if it exists.
   */
  public isComplete(sequence: Readonly<EventType[]>, groupKey?: string): bool {
    return sequence.length === 2;
  }

  /**
   * Run zero or more commands in response to a completed sequence.
   *
   * Because there is DI, one may call events with the CommandBus directly,
   * or use other side effects in response to the finished sequence,
   * and return an empty observable.
   *
   * Or alternatively, they may return a non-empty observable, similarly to a
   * saga property, and nestjs/cqrs will call all commands in the observable.
   *
   * The tracker may provide a command bus implementation, and be notified
   * about each new returned value. This would allow the tracker to f.e.
   * define that all commands in a sequence are to be executed in a transaction,
   * and do that within its command bus implementation.
   *
   * Once the returned observable completes, and all commands in it have
   * finished executing, the tracker should archive all events in the sequence,
   * and remove the group for this saga.
   *
   * A custom command bus implementation that wraps commands
   * in a transaction may commit the transaction at that point.
   * Error handling of such a wrapper is tracker dependent
   * A generic implementation might f.e. always rollback, not retry, but log the error.
   */
  public run(sequence: Readonly<EventType[]>, groupKey?: string): Observable<ICommand> {
    return of(new MyCommand(sequence[0], groupKey), new MyOtherCommand(sequence, groupKey));
  }
}

And of course, there's going to be an interface for saga trackers to implement, and a built in "in memory" tracker that doesn't persist across restarts, but still allows one to use the class, and later swap out the tracker for something that is persistent. I wanted to get this out for comments before I try to implement something that would handle this though.

I think this is simpler than the above, in that you only need to use it if you have a sequence of events you care to preserve across application restarts. If you only need to keep events not handled by all of their event handlers, what's currently in #172 is sufficient for a tracker to be created. In fact, I already have such a tracker in my app where I use cqrs as it appears in that PR.

[raymondsze]

@boenrobot
I'm not familiar with rxjs observable. Yes, what I think is like writing another library to replace nestjs/cqrs.

NestJs is great and cqrs + event-sourcing + saga seems to become a trend to solve the distributed transaction issue in microservice architecture.

I know nestjs/cqrs is just a lightweight library for doing cqrs in the nestjs framework. But, I would like to see if it is possible to bring the cqrs into production if we are using nestjs. My current view here is this is not a production library we could make use of if we really want a cqrs system in a production site.

There are mature library in Java/C# world but likes nothing in NodeJs. And I think it is good to make a cqrs framework on top of NestJs because of its dependency injection nature.

Here is what I think nestjs/cqrs is lacking of.

1. Unable to override the commandBus, the saga would inject the commandBus to execute the command. However, what we usually do is the saga should put the command to a message broker.

2. Unable to track if event / saga (may be should include command and query as well) is processed so we are unable to send back the ack to the message broker.

3. Unable to customise the event name and command name. Currently this library using the constructor name and unable to change it.

4. Need a way to carry some context throughout the command execution/event subscription. In cqrs system, we usually need correlation, causation identifiers to track what's happening. This identifier could be used to identify saga instance as well. Both eventBus and commandBus are using the interface specified by this library. The more flexible way should be.. we could allow to replace the whole eventBus or commandBus instead of implement the custom "event publisher". This is important because the command sent by saga is an asynchronous process which require us to persist the command that have been sent.

And, there are lots of stuff we could explore... like snapshotting, event upcaster, aggregate saga, saga locator, ...

[boenrobot]

1 and 3 are easy additions... a string to identify something in a decorator is a common pattern in NestJS, and the current default subscribers to the stream just need to be exposed for overriding.

2 is what this issue and the suggested interface above is for.

For 4, I don't think keeping correctional properties in events is a bad pattern. Once a generic handler for an event store and/or message broker is made, one would still reuse it for all events in the system somehow.

Oh, and... I'm not at all good with rxjs either... but that's what nestjs/cqrs uses to pass events around. I'm not sure why tbh. That's for @kamilmysliwiec to say.

[raymondsze]

@boenrobot @kamilmysliwiec
Lets say how to do followings based on current implementation.

Question 1. Where and how to persist event to event storage? This is necessary as event is a fact happened in an aggregate. You may think we could store the event into eventstore inside event handler instead of publish the event directly to eventBus. And use another way to subscribe the event storage then perform eventBus.publish (its transactional outbox pattern). But it actually doesn't work. Why? There could be multiple event handlers, if you choose to save the event in event handler, there is no way to make the event saving atomic if they are multiple events published by command handler. And, there is no way to know all the event handler (including sagas) have been executed.

Question 2. Where to persist the command triggered by sagas? We need to guarantee events can be consistently handled even the server is restarted during in the middle of the execution, this is why we need to persist the event to a event storage. Sagas are drived by event, although those commands are so-called "command", they are dispatched because of an event (which is also a fact) happened, so we need a way to persist the command as same as what we do for events. To do this, the commands have to be saved into a command storage instead of directly using commandBus to dispatch.

Question 3. How to handle the command failure that triggered by sagas? Event handler is supposed to be "fire and forget". Normally, when using sagas, we save the commands into a storage and using transaction outbox pattern again to dispatch the command to message broker, the command handler handle the command and if there is command validation exception, reply an "alert message" to message broker, the sagas subscribe the alert message and perform the compensation command. So sagas is actually just a simple event handler with the ability to dispatch command in asychronous way (fire and forget). I dont see any option to do the above based on current implementation.

Question 4. Someone may argue that sagas could subscribe the whole event stream but event handler cannot. Actually given we persist the event to event store, we won't query all the events to reconstruct the whole event stream. I dont think it would have a case "given some sequence of events happened, do some commands". Instead we should have a persistent state for sagas. The case becomes "given the current saga state is..., and a event comes, update the saga state, then do some commands...". Its also why there is a concept that sagas could be treated as special kind of aggregate, the difference is the domain object of sagas is the "intent". If you read the documentation of C# framework akka, the sagas is also called "AggregateSaga", it has its own events and all events are persistent to event storage make use of event-sourcing. There are only two difference between sagas and aggregate root.

1. Saga can apply events to construct its current state and also dispatch other domains' commands while aggregate root cna only apply events.
2. The saga identifier is identified by the events coming from other domains while the aggregate root identifier is identified by the command.
   Based on that, we need a way to store the commands and events together in the saga handler with single atomic transaction. I dont see any way to do it becuase of the "observable" implementation.

I' ve read a lots about cqrs pattern, and I don't think the current implementation provide enough way to extend to do the above. And what I think of the above is the necessary block in a cqrs system (maybe I'm wrong). If I just need a reactive stream to do something without concern of the transaction, why not I just use microservice transport currently provided by nestjs?

My current conclusion is using observable to trigger the event handler as well as the sagas could not provide a "hook" that I could persist the event or commands in single transaction and this is very important for a cqrs system. And, the offical documentation mentioned that eventBus and commandBus are oberservables so we could easily extend it to do something like event-souring... But currently, I feel the observable is actually a blocker to do event-souricng.

I actually don't see the point of using Rxjs to implement that.

This is what I think:

1. Remove the observable behviour. The event handler should be treated similary to command handle one but without "await".
2. Allow developer to implement the whole commandBus and eventBus to perform transactional outbox pattern to persist something, maybe we could use a decorator like @EventBus or @CommandBus to override the default inmemory bus.
3. Throw away @saga..., Saga could be implemented by AggregateSaga + EventHandler (Maybe we could name it as SagaEventHandler).
4. Add a AggregateSaga class with ability to dispatch command via commandBus.
5. All events and commands (from sagas) should be cached until all saga event handlers have been executed, then trigger a hook allow developer do the real publish and disptach, it would allows a chance for developer to make the transaction atomic.

[boenrobot]

Question 1. Where and how to persist event to event storage?

See #172 and the IEventDispatcher interface. The idea there is that you or a 3rd party write your own event dispatcher, possibly extending the built in one, and hook it up at onModuleInit (or else the default ones get hooked onApplicationBootsrap). The event dispatcher is responsible for persisting the events in the store, and upon hooking it up, you can also fire incomplete handlers.

Question 2. Where to persist the command triggered by sagas? We need to guarantee events can be consistently handled even the server is restarted during in the middle of the execution, this is why we need to persist the event to a event storage.

You don't need to persist the command call, you need to persist the events that form the saga, so that you can rerun the command if it stops in the middle of its execution OR in the middle of the sequence's formation. Both of those cases can be addressed on a user level with something like the above.

Just as with event handlers tracking in #172 , you or a 3rd party would need to implement an interface that is hooked into nestjs/cqrs and would handle code like the above.

Question 3. How to handle the command failure that triggered by sagas?

Good question... it seems like currently, nestjs/cqrs assumes command, event and query handlers should catch errors inside them, meaning a saga shouldn't need to handle errors.

That said, I'm not sure that would always be possible... hmm... perhaps an extra method in an interface like the above would address that?

[raymondsze]

The commad triggered by saga need to be persisted. As we rerun the saga, the command would be redisptached and results in duplicate events. So we need mechanism to deduplicate the command, check if the command is triggered by this saga before. But its not possible due to the commands are generated runtime by saga itself (i.e the command id would not be the same). So the only way is make the whole saga dispatched commands as a single transaction. If the transaction fail, no command could be saved. If the transaction succeed, all commands saved to somewhere. Use the transactional outbox pattern to fetch all these commands and publish to message broker. With this way, the command could possibly redispatched as well, but now the command id are similar because the source came from the persistent storage instead of saga runtime.
https://groups.google.com/forum/m/#!topic/dddcqrs/bOk8AbiN2BM

The book Microservice Pattern (With Java examples) (if you have bought this book) also mentioned that we need persist the commands disptached by saga using event-souricng way called SagaCreated, SagaCommand and SagaUpdated, but I think the idea is the same, and whether we should use event-sourcing to store it is optional.

Here what I want to say is, store the commands triggered by saga but not the usual commands.

So what we need to do is when saga disptach multiple commands, need a way to treat these commands as single transaction instead of treating commands each separately.

The idea of aggregate saga:
https://blog.jonathanoliver.com/cqrs-sagas-with-event-sourcing-part-i-of-ii/
and https://blog.jonathanoliver.com/sagas-event-sourcing-and-failed-commands/

Here is what we would do by referencing framework of Java and C# world as well as what the Microservice Patterns book mentioned.

1. When command comes, deduplicate the command.
2. Grab the aggregate id from command and construct back the aggregate root by fetching the historical events (For each event applied, mark the event with the aggregate version + 1)
3. execute the command handler to get uncommited events (If there is validation failure, and if the command come from saga, instead of throwing error, wrap the error reply as an event and send to message broker to notify saga)
4. save the events to event store, the (aggregateId, aggregateVersion) should be compound unique key, so if the saving fail, it means there is other command handler have been run that affect the aggregate during step1 to step3. Then we should rerun it from step1.
5. create a event-relay servcie that grab all the unpublished events and send them to message broker. Then mark these events as published.
6. events come to domain service and trigger the event handlers and saga event handlers.
7. for event handlers, just execute it as it does not require the aggregate itself but denormlize the event to the view storage that going to be qureried in the future.
8. for saga event handlers, just like command handlers, we run the saga locator to get the saga id from the event.
9. with the saga id, construct back the aggregate saga (its the saga state) by fetching historical events and apply those events (same as command handler, we need mark the aggregate version).
10. run the saga event handler to get undispatched commands and uncommited events.
11. save the commands and events to some storage with single transaction. (same here, we could use "aggregation version" to handle concurrent modification, although it should seldom happened for saga)
12. the event relay works as same as step5.
13. create a command-relay server to fetch all the commands and send them to message broker. (same as event-relay server)
14. the commands dispatched to the domain service and go back step1.

[boenrobot]

The commad triggered by saga need to be persisted. As we rerun the saga, the command would be redisptached and results in duplicate events.

This can be avoided by making the commands themselves idempotent, i.e. identify each state modification by some ID, formed by information available in the command, and persist that. And if you have all events stored, and call the same command with the same events, the same idempotency ID will be formed, but since it will already exist, the command handler being executed would detect it's duplicated and can then act differently based on that - it can either emit the old events (perhaps with an additional flag to tell handlers that this is a replay), or not emit certain/all events at all. The exact behaviour on replay and formation of the idempotency ID can be left to the command handler.

I mean, I may intentionally want to replay events... I wouldn't want the framework to stop me if that's the case. Example where you'd want to do is precisely errors in the middle of a saga - you'd want to replay all events up to that point without their side effects, and state changes, but do the remaining side effects and state changes.

[raymondsze]

@boenrobot
Yes, you could make the command issued by saga with a idempotent id. But what should be the id as these commmands are triggered by saga runtime?
What I think the possible ids could rely on the ordering that a saga issue the commmands. For example, if a saga would issue command A,B,C and given saga id is saga0. Then the command id of A, B, C could be saga0.0, saga0.1, saga0.2. So if a command comes, deduplicate the command by checking the id. But I found that persisting the message is a general practice from what I see in other frameworks. For example, https://docs.particular.net/nservicebus/sagas/, see the "Consistency considerations" session, it mentioned storing the messages prepared for transactional outbox pattern.

For replaying saga, we could just replay the events persistented for the saga if we use the event-sourcing mechanism to store the saga instance (i.e treat a saga as aggregate). No new commands or events should be persisted when replaying the events. So here we could just ignore all the saga handler when doing the replay.

What I think is saga is still something that have its internal state, and we could use something like state machine to maintain its current state and what should perform next. Thats why I think saga does not need to subscribe the whole event stream, but just one event as the current state is already persisted when previous event is already handled.

[boenrobot]

What I think is saga is still something that have its internal state, and we could use something like state machine to maintain its current state and what should perform next.

There's too much dry theory in this sentence alone, let alone your entire comments, not enough practical considerations or any implementation details for me or anyone else to consider... Please, let's talk in practical terms what do you think the framework should provide (and how) vs. what 3rd party persistent storage handlers should provide (and how) vs. what should applications provide (and how).

With that in mind...

In practice, you have a finite relatively small amount of volatile memory (RAM) available in the NodeJS process, you may or may not be hooked to a volatile storage like Redis to enable multiple node processes to share RAM data, and you most probably have some sort of larger (but still finite) non-volatile database (be it an RDBMS or a NoSQL database) where your data persists. Maybe you also have a specialized non-volatile storage for events (Kafka, EventStore, etc.), maybe you don't (and just store events in your DB).

Either way, the NodeJS code should be made so that it can recover from non-volatile sources on startup, as the process itself is volatile (e.g. if you reload due to an error or to introduce a new feature). nestjs/cqrs doesn't do that right now, and that's the part I'm onto trying to fix, and have already fixed for event handlers, just not for sagas. As @kamilmysliwiec has already stated at the start of the issue, and the reason it is labelled as "closed" is that nestjs/cqrs should not provide any persistent storage mechanism... I agree, so instead, I'm merely trying to provide extension points for 3rd parties to handle different persistent storage mechanisms.

In practice, we're also trying to allow projects to get these features with minimal to no changes of whatever they may have already developed. Issues should only arise as they opt into using a persistent storage and/or whatever new features are introduced.

We're also trying to minimize the storage (both volatile and non-volatile) that the framework and 3rd party storage handlers need to use, so that there's enough space for the application to operate. It should be the application's responsibility to decide if, when and how to archive/delete what it deems to be old data that it won't need to lookup.

I think it's fair to say that properties marked with the saga decorator, as they exist right now, are insufficient for this, and may not get the benefits of automatically being tracked by 3rd party persistent storage handlers. But beyond that, things are negotiable.

But what should be the id as these commmands are triggered by saga runtime?

Again, this is up to the command handler, not the "saga runtime" (I'm not sure if you mean the storage handler of nestjs/cqrs, but either way...). In a typical scenario, you'd pick one or more of the properties in one or more of the events to determine that.

f.e. you have an OrderCreatedEvent and PaymentReceivedEvent, correlated by "orderId". You have the saga watch out for both events, and run a command to mark the order paid if both events have occurred, regardless of the order of events. The command would then emit an event declaring that it has done that, letting further logic (notifications, etc.) take place.

For the sake of simplicity, let's assume orders can only be paid in one payment transaction. In that case, the command handler can do

const { orderId, paymentTransactionId } = command;
const outbox: IEvent[] = [];

// If this causes an exception, e.g. a connection error, no events will be published
// In this example, we're assuming a MySQL over a TypeORM repo
const updateResult = (await this.orders.update({ paymentTransactionId }, orderId)).raw;

// If the order exists, but was not modified, it means it was already paid.
// Meaning that this is a replay.
const isReplay = updateResult.changedRows === 0 && updateResult.affectedRows === 1;

// We could've not send an event at all, but for this example,
// we're sending one with a flag, allowing the consumer to decide
// how to handle the fact that this is a replay.
outbox.push(new OrderPaidEvent({ orderId, paymentTransactionId, isReplay }));

this.eventPublisher.publishAll(outbox);

In a more complicated example, where the order might be paid over several payment transactions, you'd need to use the paymentTransactionId as the idempotency identifier in a separate table, keeping track of all payments for an order, and maybe emit a different event for each transaction, maybe have the saga check the sum, and consider the order paid if it's >= the created order amount and do a return of the difference if there's any over.

"But what about if we have to come up with a new idempotency ID during the saga" I hear you. If you need to make such IDs, you'd be making them in response to those events, so by then, you can check if you already have an ID generated or not.

f.e. in the above scenario, if an OrderPaidEvent event handler wanted to send back a new ID, acknowledging to a payment provider that a payment was processed, it would first lookup if it has already made its own ID into the DB in response to that paymentTransactionId. If not, generate one, save it, send it, and emit an event to tell the rest of the system that the ID was created, and sent. If it has an ID already, send what it has created previously, and only emit an event about it being sent (with a replay flag).

The generated ID doesn't need to be random (only unique), so in the case of f.e. a partial refund (in the above "more complicated scenario"), where there's a new payment transaction created on the app end first, the idempotency ID might be formed by a hash of all paymentTransactionId values that form the otherwise overpaid order, up to the first overpaid transaction, and for each further transaction (now assumed to be an overpaid transaction), base the ID based on the paymentTransactionId alone.

Thats why I think saga does not need to subscribe the whole event stream, but just one event as the current state is already persisted when previous event is already handled.

If you only need to monitor one event, and you don't care about prior events, you may as well use an event handler. A saga is needed when you have multiple related events, and you only need to do something if all related events have been done prior to whatever is the latest related event.

Without a saga, the workaround would be to have a handler listening for all event types, and do essentially what a saga does for you within it - lookup related events, and only do a thing once all related events are in storage, and the needed info is available. That part is cumbersome, and yet generic enough to be handled by the framework, instead of such a generic event handler.

What I think the possible ids could rely on the ordering that a saga issue the commmands. For example, if a saga would issue command A,B,C and given saga id is saga0. Then the command id of A, B, C could be saga0.0, saga0.1, saga0.2. So if a command comes, deduplicate the command by checking the id.

In practical terms, this would mean that each saga would also need to maintain a sequence ID that is also in persistent storage, and is generated for each new sequence, plus a command offset, also in persistent storage... and also keep a status for each command in persistent storage, so that only non-started and unfinished (and ones that threw exceptions?) could be started on startup... Which seems like a lot of wasted space, when you could use idempotency IDs like above, and in some applications, you might need to use such anyway.

But fair enough, if a persistent handler wants to implement that, I guess they can do that with a persistent handler around a saga class like the above, no? Like, before every run() (i.e. after the isComplete() returns true; the persistence handler will be notified of that), make a new sequence ID based on a hash of all events in the event sequence. For each emit of the returned value, add an offset and set status as not started, along with a name and the object itself. In the saga command bus, attach a then().catch() to the returned promise that would update the status to success or failure, and once the stream completes, maybe also mark the command sequence itself as completed, so that you know the command sequence didn't fail in the middle of adding commands.

Then on startup, get all sequences containing unfinished commands, and for each sequence, set the sequence ID to the persistent handler's command bus, and run the commands, re instantiated from the name and object previously saved.

A lot of wasted space, but doable with a handler for the above saga class (implemented by persistent handlers, called by nestjs/cqrs).

[marcus-sa]

@raymondsze @boenrobot a long nice conversation.
Did you ever get to implementing any of this or writing a specification?

[boenrobot]

This previous comment is the closest thing to a spec I can muster.

I haven't implemented it yet, mostly because the project I needed this for got canceled (due to our team lacking an experienced front ender), and I don't have enough free time to invest in it.