parumu / mq

Message Queue

Message queue service with the following characteristics:

• Each topic has an associated fixed size queue. The queue evicts older messages to store messages in the fixed size queue.
• Topic/owner/subscriber name has a length limit
• There is a message size limit
• There is a limit for maximum number of topics
• There is a limit for maximum number of subscribers

Configuration

• Files
  • src/main/resources/log4j.properties
  • src/main/resources/config.properties
    • Configurable keys and the examples:

# address that Message queue service binds to
host = 0.0.0.0 

# port that Message queue service listens to
port = 8080

# Redis host and port
redisHost = localhost
redisPort = 6379

maxQueueSize = 10000
maxNameLen = 100

# 128KB = 128 * 1024 = 131072
maxMsgSize = 131072

maxTopics = 10000
maxSubscribers = 10000

• Redis
  • Should set maxmemory-policy: noeviction to avoid Redis to evicts data from memory.

Building a docker image

Run docker build command in the project root directory. e.g.

$ cd [project root directory]
$ docker build -t mq .

Running tests

Tests in test directory contain unit and integration tests. In order to run integration part of the tests, Redis needs to be running locally. To do so:

$ cd [project root directory]
$ docker-compose up

Then below should run both of the unit and integration tests

$ mvn test

API

Endpoint	Request	Result	Description
v1/topic/register	"owner": string "topic": string	200 success 400 error	Register a topic specifying the owner
v1/message/publish	"owner": string "topic": string "msg": string	200 success 400 error	Publish a message to a topic specifying the owner
v1/topic/subscribe	"subscriber": string "topic": string	200 success 400 error	Let a user subscribe to a topic
v1/message/get	"subscriber": string "topic": string	200 success msgIdx: int msg: string 230 message evicted 231 no message 400 error	Let a user retrieve the latest unacked message in a topic
v1/message/ack	"subscriber": string "topic": string "msgIdx":number	200 success 400 error	Let a user ack a message of a topic

Data Structure

Ledends: <X>: actual value for X
--
num-topics -> # of topics

T-topic -> linked list of messages for topic T e.g. ["msg1", "msg2", .... "msgN"]

T-topic-attr
  - owner -> <topic owner>
  - msg-head -> index of the message at the beginnig of the topic list (message index is 1-based)
  - num-subscs -> number of subscribers

T-topic-subsc
  - <subscriber> -> next message index to get of the subscriber

• Register API call newly stores T-topic-attr/owner, T-topic/attr/msg-head=0 and T-topic/attr/num-subscs=0 for the topic.
• Subscribe API call sets T-topic-subsc- to T-topic-attr/msg-head + 1
• Publish API call adds a message in front of T-topic list and increase T-topic/attr/msg-head by 1. If T-topic list is full, the message at the end is evicted before adding the new massage.
• Get API call attempts to get a message at T-topic-subsc-. If the message at the index has not been published, the subscriber gets 231 no message. If the message at the index has been evicted from the list, the subscriber gets 230 message evicted.
• Ack API call acks a message at T-topic-subsc-, and increases T-topic-subsc- by 1.

Runtime complexity

• Get: O(1) to O(n)
  • Messages are stored in linked list. Getting message at the beggining and end is O(1) and O(n) respectively.
• Others: O(1)

Maximum memory usage

• Maximum memory usage by the configuration can be rougnly calculated with the following formula:

lists = maxMsgSize * maxQueueSize * maxTopics
topics = maxNameLen * maxTopics
owners = maxNameLen * maxTopics
subscribers = subscriber * maxSubscribers * maxTopics

maxMemoryUsage = lists + topics + owners + subscribers