Jump — Message Queuing Demo Project

One running system, one RabbitMQ broker, and every core message-queuing concept demonstrated by a different piece of it. Use this alongside the class plan — point at a different service depending on which concept you're teaching.

Setup

# 1. Start RabbitMQ (broker used by every service)
docker-compose up -d

# 2. Confirm it's up - management UI at:
#    http://localhost:15672  (user: guest / pass: guest)

# 3. Install Python deps
uv sync

Project Structure

jump/
├── docker-compose.yml            # single RabbitMQ broker, used by all services
├── common/
│   └── rabbitmq_config.py        # shared connection + topology settings
├── order-service/                # PRODUCER (persistent msgs + publisher confirms)
├── email-service/                # CONSUMER - pub/sub + idempotency (at-least-once)
├── inventory-service/            # CONSUMER - pub/sub + retries + app-level DLQ
├── payment-service/              # CONSUMER - native DLX + basic_nack(requeue=False)
├── analytics-service/            # CONSUMER - pub/sub (simplest subscriber)
├── at-most-once-consumer/        # CONSUMER - auto_ack (message loss on crash)
├── image-worker/                 # CONSUMER pool - work queue / point-to-point
├── routing-demo/                 # topic + direct exchange routing
├── load_generator.py             # floods the work queue - backpressure demo
├── worker_autoscaler.py          # optional - scale image-workers from queue depth
└── message-queuing-explainer.ipynb

Concept → Where To Look

Concept	Run this	What to point out
Producer/consumer basics	`order-service` + any one consumer	One publishes, one receives — decoupled, don't block each other
Pub/sub	`email-service`, `inventory-service`, `analytics-service` together	Each has its OWN queue bound to the same fanout exchange — all three get every event
Point-to-point / work queue	Two or more `image-worker` instances	Jobs get round-robined — each job goes to exactly ONE worker, not all of them
At-least-once + idempotency	`email-service`	Ctrl+C mid-"send", restart — same order redelivered; idempotency skips duplicates
At-most-once / message loss	`at-most-once-consumer`	`auto_ack=True` — Ctrl+C mid-processing and the message is gone forever
App-level DLQ	`inventory-service`	After retries, service republishes to `inventory_dlq` itself
Native DLX + nack	`payment-service`	Queue has `x-dead-letter-exchange`; `basic_nack(requeue=False)` lets the broker route to `payment_dlq`
Persistence + confirms	`order-service`	`delivery_mode=2` + `confirm_delivery()` — durable queue alone is not enough
Topic / direct routing	`routing-demo`	Same events, different binding keys — who receives what
Backpressure	One `image-worker` + `load_generator.py`	Queue depth climbs in the management UI
Autoscaling	`worker_autoscaler.py` + `load_generator.py`	Starts/stops `image-worker` processes from ready depth

Suggested Live Run-Through

Open several terminals:

# Terminal 1 - analytics subscriber
cd analytics-service && python analytics_service.py

# Terminal 2 - email subscriber (idempotency demo target)
cd email-service && python email_service.py

# Terminal 3 - inventory subscriber (app-level DLQ)
cd inventory-service && python inventory_service.py

# Terminal 4 - payment subscriber (native DLX + nack)
cd payment-service && python payment_service.py

# Terminal 5 - one image worker
cd image-worker && python image_worker.py worker-1

# Terminal 6 - place some orders
cd order-service && python order_service.py 5

Watch the pub/sub services print the same order events independently. Watch image-worker pick up thumbnail jobs one at a time. Watch some inventory/payment failures retry; exhausted ones land in inventory_dlq (manual) vs payment_dlq (broker DLX).

Idempotency (at-least-once): while email-service is mid-send (2s sleep), Ctrl+C, restart — unacked message is redelivered.

At-most-once contrast:

cd at-most-once-consumer && python at_most_once_consumer.py
# place an order, Ctrl+C during the 3s sleep, restart — that order never comes back

Topic / direct routing:

# three terminals
cd routing-demo && python routing_consumer.py topic-all
cd routing-demo && python routing_consumer.py topic-us-ship
cd routing-demo && python routing_consumer.py direct-shipped-us

# then publish
cd routing-demo && python routing_publisher.py

topic-all (order.#) gets everything. topic-us-ship and direct-shipped-us only get order.shipped.us.

Backpressure:

# with only worker-1 running
python load_generator.py 50

Then start more workers, or use the autoscaler:

python worker_autoscaler.py
# other terminal:
python load_generator.py 50

Unacked messages — when do they return to the queue?

With manual ack (most services here):

Broker delivers the message → it moves from Ready to Unacked.
It stays reserved for that consumer until one of:
- basic_ack → deleted (done)
- basic_nack / reject with requeue=True → back to Ready
- basic_nack / reject with requeue=False → discarded or dead-lettered
- consumer channel/connection closes (crash, Ctrl+C, network drop) → broker automatically requeues remaining unacked messages
There is no “are you done yet?” poll. Redelivery is driven by ack/nack or channel death. (RabbitMQ also has a long consumer_timeout; if you never ack for a very long time the connection can be closed, which then requeues — still not a gentle retry timer.)

With auto_ack=True (at-most-once-consumer): the broker forgets the message at delivery time. Crash mid-processing = permanent loss.

Notes for Teaching

Everything uses one RabbitMQ instance — patterns differ by how you set up exchanges/queues/consumers, not by spinning up extra brokers.
email-service's idempotency store is an in-memory set for demo only; production needs Redis / a DB unique constraint so it survives restarts.
Durable queue ≠ durable message. Persistence needs delivery_mode=2 (and usually publisher confirms) as well.
inventory-service vs payment-service is intentional: same problem (poison messages), two approaches (app republish vs broker DLX).
Kafka consumer-group comparison remains a natural stretch goal.

solisoma/RabbitMQ-Tutorial

README