narayanbhat166 / leetcode_datahouse Goto Github PK

Cargo workspaces provide a neat way to manage multiple libraries and binaries which share utilities into a single repository. More information can be found here The many binaries that will be in this project are

• controller: A controller is responsible for coordinating resources between multiple producers. It should implement a efficient way to buffer the data sent by producers. Provide a layer of abstraction around message queues for producers and consumers.

• producer: Scrape data from leetcode and send it to controller. This can be scaled very effeciently.

• consumer: Flush the data from message queues to database and file storage.

The multiple libraries that will be included in this project are

• Redis: Provide redis helper functions to communicate to redis.
• Kafka: Provide kafka helper functions to communicate to kafka brokers.
• Configs: To manage configs.
• protobuff: To compile proto files to rust specific structures and functions.

Since this would be a large refactor, I will be breaking this down to child issues

• #2
• #3
• #4

A dashboard which will provide near real time data extracted by the producer and total data consumed by the consumer.