andyg7 / friendsdrinksevents

FriendsDrinksEvents

Technologies

Gradle

Used to build the application JAR.

Kafka and Kafka Streams

Kafka Streams is used to build multiple asynchronous "micro-services". See code in https://github.com/andyg7/FriendsDrinksEvents/tree/master/src/main/java/andrewgrant/friendsdrinks - any class that ends in Service contains a main method that starts a Kafka Streams application. A REST API fronts interacting with the backend stream services - see https://github.com/andyg7/FriendsDrinksEvents/tree/master/src/main/java/andrewgrant/friendsdrinks/frontend. The REST API uses the Producer and Consumer clients to interact with the Kafka Streams app(s). The REST API also runs it's own Kafka Streams application for creating some materialized views. The API endpoints are consumed by https://github.com/andyg7/FriendsDrinksUI which ultimately is how the end user uses this app.

Resources

• https://kafka.apache.org/20/documentation/streams/developer-guide/dsl-api.html
• https://docs.confluent.io/current/streams/architecture.html

Docker

Used to build the application's Docker image. The same image is used to run the different services because one single jar is built at compile time. Thus in Kuberentes it's expected the class to run is provided as an arg when the image is used to run a container.

Kubernetes

Used for running the various containers that make up the application. There are various Kubernetes resources (e.g. StatefulSet, Service, ConfigMap etc.) for running ZooKeeper, a schema registry, a Kafka cluster and then the actual streaming application. See https://github.com/andyg7/FriendsDrinksEvents/tree/master/kubernetes.

Semaphore

CI/CD solution. There's a pipeline that builds the code and then deploys the app to AWS EKS (managed Kubernetes). https://github.com/andyg7/FriendsDrinksInfrastructure contains yet another pipeline that sets all of this AWS infrastructure up. This includes creating a VPC, EC2 instances, security groups, IAM roles, the actual Kubernetes cluster etc.

friendsdrinksevents's Issues

Refactor tests to not depend on running kafka container

Refactor tests to not depend on running kafka container

Fix REQUESTS_STORE so it has delete requests

Fix REQUESTS_STORE so it has delete requests, not just create requests

Upgrade to latest Kafka Streams version

Upgrade to latest Kafka Streams version https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams

Support password during user creation

Move avro package out of src directory

Move avro package out of src directory

Write some unit tests

Write some unit tests

Dockerify different services

Create separate docker-compose file for stream services

See https://florimond.dev/blog/articles/2018/09/building-a-streaming-fraud-detection-system-with-kafka-and-python/ and https://docs.docker.com/compose/networking/ as references

Create service for updating topic that stores current friends drinks

Initial Frontend REST API for interacting with backend streams

Give each service its own application id

Create serializers for id classes

Integrate with AWS Cognito

Integrate with AWS Cognito for user management.

API for interacting with friends drinks

Refactor User avro

Refactor User avro