📨 Message Queues

Middleware that stores and forwards messages.

Producer → Queue → Consumer

Producer sends message.
Queue holds message.
Consumer processes when ready.

Decoupled, asynchronous communication.

Service A calls Service B directly:

A: "Process this order"
B: (processing... takes some time...)
A: (waiting... waiting...)
B: "Done"
A: (finally continues)

Synchronous. A is blocked.

A → Queue → B

A: "Here's an order" (sends to queue, moves on)
A: (continues doing other work)
B: (picks up message when ready)
B: (processes at its own pace)

Asynchronous. A is free.

User uploads image.
Resize needed (slow!).

Without queue:
  User waits... waits... waits...

With queue:
  "Image queued for processing"
  User continues.
  Processing happens in background.

Order service needs:
  - Inventory update
  - Email notification
  - Analytics tracking

Direct calls: Order service knows all three.
Queue: Order publishes "OrderPlaced". Others listen.

1000 requests/second spike!

Without queue: Services overwhelmed.
With queue: Messages buffered, processed at steady rate.

Sends messages to queue.
"I have work for someone."

Receives messages from queue.
Processes the work.

Holds messages.
FIFO usually (first in, first out).
Persisted until consumed.

The data being sent.
Usually JSON, but can be any format.

{
  "type": "order",
  "orderId": "123",
  "userId": "456"
}

One producer, one consumer.
Message delivered to exactly one consumer.

Producer → Queue → Consumer

Work distribution, task processing.

One producer, many consumers.
All subscribers get the message.

Producer → Topic → Consumer 1
                → Consumer 2
                → Consumer 3

Events, notifications, broadcasting.

Multiple producers, multiple consumers.
Messages distributed among consumers.

Producer 1 ─┐
Producer 2 ─┼──→ Queue ──→ Consumer 1
Producer 3 ─┘          ──→ Consumer 2

Scale processing by adding consumers.

Message delivered 0 or 1 times.
May be lost if consumer fails.
Fastest.
Use when: Logging, metrics (loss acceptable)

Message delivered 1 or more times.
Typically not lost, but duplicates are possible.
Consumer must handle duplicates.
Use when: Most applications

Message delivered exactly 1 time.
Complex to implement.
Performance overhead.
Use when: Financial transactions

How does queue know message was processed?

1. Consumer receives message
2. Consumer processes message
3. Consumer sends ACK (acknowledgment)
4. Queue removes message

No ACK? Message redelivered.

If consumer dies before ACK:
  Message goes to another consumer.
  Message can be retried.

Messages that can't be processed.

Failed 3 times?
→ Move to dead letter queue.
→ Investigate later.
→ Doesn't block main queue.

Handle edge cases gracefully.

RabbitMQ:
  Complex routing, traditional messaging

Kafka:
  High volume, event streaming, replay needed

SQS:
  AWS environment, simplicity, serverless

Redis:
  Speed priority, simple queuing

Processing fails.
Wait a bit, retry.
Fails again? Wait longer.
Increase the delay over time.

Exponential backoff prevents hammering.

VIP orders processed first.
Emergency alerts skip the line.

Priority levels determine order.

Message older than a chosen window?
Discard it.

Time-sensitive data doesn't wait forever.

At-least-once means duplicates possible.
Consumer must be idempotent!

Process order → Check if already processed.

Bad message blocks queue.
Processing stuck forever.

In many systems, setting up a DLQ (or equivalent) is a good idea.

Very large messages → Slow, memory issues.

Better: Store data elsewhere, send reference.
"Process file at s3://bucket/file.jpg"

Messages may arrive out of order.
If order matters, use partitioning or sequence numbers.