Aggregate Not Found when AGGREGATE_CACHE_MAXSIZE not set about eventsourcing

[darrenallen85]

Will close this as when i restructured and cleaned up the code I did not have the issue anymore 😮

[johnbywater]

I'm glad you got it working! I don't know if you have changed this, but I would make sure you call the application save() method at most once when responding to a client request.

[johnbywater]

Thanks for your interest in this library. If you have any other issues, please don't hesitate to post again.

[darrenallen85]

Thanks for the quick reply. When you say at most once ... you mean at most once per aggregate yes ?

[johnbywater]

Once per request. And if necessary save multiple aggregates in the same call to save().

https://eventsourcing.readthedocs.io/en/stable/topics/application.html#saving-multiple-aggregates

[darrenallen85]

Thanks for that.

1. On another note I was reading [https://eventsourcing.readthedocs.io/en/stable/topics/system.html] to try and undersand how to have one application respond to the domain events of another app. (https://eventsourcing.readthedocs.io/en/stable/topics/system.html) - i didnt quite understand but found some explanations here here - are the explanations still relevant ?

2. In our case we were wanting the domain events to be published via an event broker - maybe kafka ... possible ? Looks like i would have to override the notify() method on the application to publish to kafka ?

3. What would be the approach in the following app to consume the domain events from kafka ?

many thanks

[johnbywater]

are the explanations still relevant ?

Yes, these are very good explanations, and still completely relevant. I just haven't completed this area of the docs in version 9 of the library. The motivation, concepts, and principles are exactly the same. The code however is slightly different. And the only reason for not writing this area of the docs is because I have been concentrating on the persistence, domain, and application modules.

You could override the application _notify() method. Alternatively you could write a "gateway" application that pulls from the notification log of your core event-sourced application, and thereby tracks what has been successfully pushed/published into Kafka (or whatever event/message broker you want to use). That way, if the event/message broker is down for some reason, or happens to fail once or twice, you can resume propagating the state of your application (the events) later, and the application can continue without presenting these errors to users.

This is an area that would usefully have some more examples, showing how to do this. I have just been focussing on the core capabilities of the library. When we get into the "event-driven" side of things, there are just so many possibilities, it isn't possible for this little library to be comprehensive. But the principle of avoiding "dual writing" in the production and consumption of events in an event-driven system, of pulling from a notification log (a.k.a the outbox pattern) is the one important idea to incorporate into your design.

[johnbywater]

PS If you are interested in this area, I would read all of these docs from v8:

• https://eventsourcing.readthedocs.io/en/v8.3.0/topics/notifications.html
  • Especially https://eventsourcing.readthedocs.io/en/v8.3.0/topics/notifications.html#three-options
  • The "big array" thing doesn't appear in v9, but local and remote notification logs do.
• https://eventsourcing.readthedocs.io/en/v8.3.0/topics/projections.html#reading-event-notifications
  • The "subscribe_to" decorator doesn't appear in v9, but the other things on this page are still supported in v9.
• https://eventsourcing.readthedocs.io/en/v8.3.0/topics/process.html
  • The single- and multi-threaded runners exist in v9. The pipeline stuff doesn't, but I'm hoping to do something about partitioning/sharding the state of an application, and system of applications, in v9. I just wanted to think a bit more about what might be useful here, since there are several options, and I'm reluctant to generate a lot of different ways of doing things (for reasons of maintenance complexity and cost). The other runners (Thespian, Ray, and gRPC) don't appear in v9. But Ray is a good tool. The gRPC runner worked very well, and I plan to create an extension package that includes this code. Maybe there will be an extension package for Ray too. And for Kafka :)

Hope that helps?

[johnbywater]

By the way, if you search in the v9 docs for the term "dual writing" you will see some of the above discussion has been carried forward in v9 docs. I do want to replicate the v8 sections I linked above, I just didn't do this yet. The considerations haven't been deprecated, just some of the extraneous complexity and multiple offerings have, and then mostly for reasons of simplicity and approachability.

https://eventsourcing.readthedocs.io/en/v9.2.2/search.html?q=dual

The difficulty in the system module is that the number of possibilities explodes. There are different ways of doing the persistence module, but there are some particular requirements for event sourcing that make it a relatively specific thing. There are more options for writing an event-sourced model, and the library domain module offers a concise way to do it, that works well, whilst affording alternatives. The application is quite straightforward combination of persistence and domain, so whilst there are different ways of doing things, the library offers a sensible way forward.

However, when it comes to the "event-driven system" side of things, firstly it is a little bit beyond the scope of "event sourcing" as such, and secondly there are just an extraordinary number of possibilities, without any one being obviously "the one way" to do it. There are also a lot of different tools, such as message brokers, microservices frameworks, orchestration layers, and cloud-native services. It really depends on the situation, and there's no hope that a little library like this can function as a guide to all of these things.

To keep things simple, what I can say is that in case you want/need the state of your distributed system to be reliable, then you will do well to avoid "dual writing" in the production and consumption of event notifications, by having "leaders" commit event notifications atomically along with new state, and by having "followers" then counting along this sequence and commit tracking records along with new state, and by giving the events of an application numbered serial ordering so that the state of the application can be propagated and processed as a sequence using counting. You don't always need to do that, but if you don't then when you system crashes or is simply stopped and resumed, you can't be sure that it will be in the state you had designed it to be in. Which, of course, might not be a problem.

[darrenallen85]

Thank you so much... I will read these docs in the next cpl of days.
Is the slack channel still active ? if so can i get an invite ?

[darrenallen85]

alternatively if you are happy with me keeping the comments here - i'm good with that too

BTW Really like the structure of the V9 library very clean and intuitive.

[johnbywater]

We can chat here if you prefer. I'll reopen the ticket!

Yes Slack workspace is quite active. There's an invite link in the docs in several places:
https://eventsourcing.readthedocs.io/en/stable/topics/support.html#community-support

[johnbywater]

BTW Really like the structure of the V9 library very clean and intuitive

Thanks! That means a lot.