suadev / microservices-change-data-capture-with-debezium

Microservices data exchange with change data capture and outbox pattern.

This simple project demonstrates how to manage eventual consistency between microservices with Change Data Capture and Outbox Pattern using Debezium, Kafka, and Kafka Connect.

Prerequisites

• .NET 5.0 SDK
• Docker Desktop

Run in Debug Mode

• Run 'docker-compose up' and wait for all infra to up and running.
• Select 'All' debug option and start debugging. (for vs code)
• Wait until all microservices are up and running.

Initiating Databases: Each service will be created its own database while it's starting for the first time.

Register Debezium Postgres Connectors to Kafka Connect

You need to register two Postgres Connectors. One for Customer Database and the other for Identity Database.

customer_config.json -> Customer Connector Config.

identity_config.json -> Identity Connector Config.

Use customer.sh and identity.sh to create/update/delete connectors. For instance, to create customer connector;

.\customer.sh create_connector

update_connector function is commented out. If you want to update the connector config, uncomment the function and download jq from here.

After registeration of two Debezium Connectors, two workers will be created on Kafka Connect which are listening to the outbox tables to push events to Kafka topics.

Check the connector list via the following endpoint and see the following json result to be sure everything is okay.

curl -X GET http://localhost:8083/connectors 

["identity_outbox_connector", "customer_outbox_connector"]

Now you are ready to test. See sample postman requests here.

Overall Architecture

When a new user created on Identity Service, eventual consistency will be obtained for Customer and Notification Services as shown following flow.

Tool Set

• Asp.Net 5.0
• Entity Framework Core 5.0
• PostgreSQL - Npgsql
• MediatR
• Kafka - Zookeeper
• Confluent.Kafka
• Kafka Connect
• Debezium
• Kafdrop
• Docker - Docker Compose
• Azure Data Studio
• VS Code

microservices-change-data-capture-with-debezium's Issues

I get topic not available error.

Hi Suat,
Thank you for this useful code that implements CDC.
However, I ran into a few issues when implementing it on my own.
First off all,
1-I used the same docker-compose file as yours.

2-However, when creating the connectors, I needed to include a new line in the config files as it would refuse to create the connectors.
What I included is the "topic.prefix" element because the I got the error saying that element was missing from the config file.
I set the "topic.prefix" to "customer" and "identity" respectively for the two conffig files.
That way I could create the connectors which I then made sure they existed.
3- I also noticed that when I ran the 3 apis, the migrations of the dbs did not take place so I manually ran "update-database" command to create the databases and their tables.
4- While running the apis, I had this exception about kafka topics which, however, did not stop api to proceed to be in the running state though.
The error I was getting was about the topic names that we are registering in the startup classes.
For instance, for the Customer api: The error reads like the following:

Confluent.Kafka.ConsumeException: Subscribed topic not available: user_events: Broker: Unknown topic or partition
at Confluent.Kafka.Consumer2.Consume(Int32 millisecondsTimeout) at Shared.Kafka.Consumer.BackGroundKafkaConsumer2.ConsumeTopic(CancellationToken stoppingToken)

For the other apis, similar exceptions related to their topics registered in Kafka Consumer service.
I wonder, if you could give me a clue as to the solution. It seems, the topics needed to be registered beforehand with Kafka. There is no automatic registration happening.

Thanks in advance.

database setup - tables/schema and data

Could you please add a short description on how to initiate the database setup. I guess it's through EF core migration.