shijuvar / go-distributed-sys

Guidance for building event-driven distributed systems and microservices in Go with NATS JetStream, gRPC and CockroachDB

Guidance for building distributed systems and Microservices in Go

Articles

• Building Event-Driven Distributed Systems in Go with gRPC, NATS JetStream and CockroachDB

Technologies Used:

• Go
• NATS, NATS JetStream, NATS Micro
• gRPC
• CockroachDB
• Go kit
• Zipkin (for distributed tracing)

Compile Proto files

Run the command below from the eventstream directory:

protoc eventstore/*.proto \
		--go_out=. \
		--go-grpc_out=. \
		--go_opt=paths=source_relative \
		--go-grpc_opt=paths=source_relative \
		--proto_path=.

Set up CockroachDB

Set up CockroachDB Cluster with three nodes

cockroach start \
--insecure \
--store=orders-1 \
--listen-addr=localhost:26257 \
--http-addr=localhost:8080 \
--join=localhost:26257,localhost:26258,localhost:26259 \
--background

cockroach start \
--insecure \
--store=orders-2 \
--listen-addr=localhost:26258 \
--http-addr=localhost:8081 \
--join=localhost:26257,localhost:26258,localhost:26259 \
--background

cockroach start \
--insecure \
--store=orders-3 \
--listen-addr=localhost:26259 \
--http-addr=localhost:8082 \
--join=localhost:26257,localhost:26258,localhost:26259 \
--background

cockroach init command to perform a one-time initialization of the cluster

cockroach init --insecure --host=localhost:26257

Start a SQL Shell for CockroachDB:

cockroach sql --insecure --host=localhost:26257

Create user

cockroach user set shijuvar --insecure

Create Databases

cockroach sql --insecure -e 'CREATE DATABASE eventstoredb'

cockroach sql --insecure -e 'CREATE DATABASE ordersdb'

Grant privileges to the shijuvar user

cockroach sql --insecure -e 'GRANT ALL ON DATABASE ordersdb TO shijuvar'

cockroach sql --insecure -e 'GRANT ALL ON DATABASE eventstoredb TO shijuvar'

Run NATS JetStream Server

nats-server -js

Run JetStream with config file

nats-server -c js.conf

// js.conf file

jetstream {
    // jetstream data will be in /data/jetstream-server/jetstream
    store_dir: "/data/jetstream-server"

    // 1GB
    max_memory_store: 1073741824

    // 10GB
    max_file_store: 10737418240
}

http_port: 8222

Prerequisites for running the eventstream demo

• Run the NATS JetStream Server
• Run CockroachDB
• Run bootstrapper app (eventstream/bootstrapper) as a one time activity for creating stream in JetStream and create two databases in CockroachDB
• In order to use distributed tracing with gokit-ordersvc (order service with Go kit), run Zipkin distributed tracing system

Basic Workflow in the example (eventstream directory):

1. A client app post an Order to an HTTP API (ordersvc)
2. An HTTP API (ordersvc) receives the order, then executes a command onto Event Store, which is an immutable log of events of domain events, to create an event via its gRPC API (eventstoresvc).
3. The Event Store API executes the command and then publishes an event "ORDERS.created" to NATS JetStream server to let other services know that a domain event is created.
4. The Payment worker (paymentworker) subscribes the event "ORDERS.created", then make the payment, and then create another event "ORDERS.paymentdebited" via Event Store API.
5. The Event Store API executes a command onto Event Store to create an event "ORDERS.paymentdebited" and publishes an event to NATS JetStream server to let other services know that the payment has been debited.
6. The Query synchronising worker (querymodelworker) subscribes the event "ORDERS.created" that synchronise the query data model to provide state of the aggregates for query views.
7. The review worker (reviewworker) subscribes the event "ORDERS.paymentdebited" and finally approves the order, and then create another event "ORDERS.approved" via Event Store API.
8. A Saga coordinator manages the distributed transactions and makes void transactions on failures (to be implemented)

Training and Consulting in India

As a Consulting Solutions Architect, I do provide training and consulting on Go programming language and distributed systems architectures, in India.

go-distributed-sys's Issues

Event Retrieval and Domain Logic Missing

Firstly, thanks for the example and the medium article. I am busy trying to make my own CQRS ES system with go. I have done work on a CQRS ES system in C# but have decided to try learn go and see if it works better.

I see that you do have the function to retrieve events for an aggregate but it is not implemented (not a problem of course)

My main issue is knowing where and how to implement the domain rules. From my experience (which may be wrong), if a command comes in then all the events related to that aggregate ID need to be retrieved and replayed to create an aggregate state. The command can then execute to see if it passes the domain logic. Like to to stop a payment if a certain order has already had its payment executed.

Once the domain logic has passed then the event can be stored and published.

Hope I am on the right track and have not been misled. I would like to know how you would do this. Or if you have a more full example for me.

Possible data inconsistency

From what I can see while reading the code there is a possibility that the system can be in an inconsistent state in a scenario when publishEvent fails to execute when creating order. I cannot see any compensating actions that handles this situation.

NATS Util package is missing

cluster discovery

am a bit confused about how each server finds the others int eh cluster.

DO you run Consul ?

Benthos for reducing code

nice example of CQRS . Thank you for making this available to everyone and writing about it on Medium.

Lately I have started using benthos to configure these types of systems.
As i read the article I kept thinking of how all the data mapping and logic could be done in benthos.

NATS Jetstream is supported by benthos.

It now also has CueLang support but it’s not finished yet .

https://www.benthos.dev/docs/configuration/using_cue

Cuelang is strongly typed to a degree and you can model your types and generate protocol buffers and open api with it.

The cool thing with cuelang is that it has modules just like golang and so you can built up a large system with it to do the streaming transformations as well as validation as it has expressions .

mit there is not enough then you can extend cuelang with golang modules , and the cuelang can use it.