Track 3: communication-patternsID: message-queue-pubsub-design

Mode: structured•Difficulty: intermediate•⏱️ 20 mins

Module · Communication Patterns · 20 min

Message Queue vs Pub-Sub — Event-Driven Microservices & Fan-Out

In large-scale distributed systems, microservices must never call each other directly for asynchronous workflows. Master the difference between Point-to-Point Queues (1-to-1) and Publish-Subscribe Topics (1-to-Many Fan-Out), and evaluate the trade-offs of RabbitMQ vs Apache Kafka vs AWS SQS/SNS.

01 / Concept Cards (Theory)

Point-to-Point vs Publish-Subscribe Mechanics

Understand how asynchronous message passing decouples services, eliminates tight runtime dependencies, and enables 1-to-many event broadcasting.

1. The Two Core Messaging Paradigms

Whenever a service needs to pass data asynchronously, it uses either a Message Queue (Point-to-Point) or a Pub/Sub Topic (Broadcast / Fan-Out):

📬 1. Point-to-Point Queue (Competing Consumers)1 Message ➔ Exactly 1 Worker

Analogy: The Bank Teller Ticket Line. Even if 10 teller windows are open, each customer ticket is served by exactly one teller. Once processed, the ticket is destroyed.

Use case: Heavy background worker tasks (e.g. video encoding, PDF invoice rendering, sending a password reset email).

📢 2. Publish-Subscribe Topic (Fan-Out Broadcast)1 Event ➔ Multiple Independent Subscribed Services

Analogy: The Newspaper / Podcast Subscription. The creator publishes one edition, and every subscriber receives their own copy simultaneously.

Use case: Domain events (e.g. OrderPlacedEvent) consumed by Billing, Inventory, Fraud Detection, and Analytics concurrently!

2. Broker Battleground: RabbitMQ vs Apache Kafka vs AWS SQS/SNS vs Redis Streams

Choosing the right message broker is one of the most critical decisions in system design. The comparison below breaks down their architectural philosophy:

Broker	Architecture Model	Message Retention	Ordering & Replay	Best Fit
RabbitMQ	Smart Broker / Dumb Consumer	Transient — Messages deleted immediately once acknowledged (ACK).	Per-queue FIFO; No replay once consumed.	Complex routing keys, AMQP headers, task queues, immediate transactional delivery.
Apache Kafka	Dumb Broker / Smart Consumer	Persistent — Append-only commit log retained on disk for days/weeks.	Strict FIFO per Partition Key; Full Replay supported.	High-throughput event streaming (1M+ msg/sec), clickstream analytics, event sourcing, CDC.
AWS SNS + SQS	Cloud Managed Fanout	SNS: Instant push; SQS: Up to 14 days in queue.	Standard: Best-effort; SQS FIFO: Strict ordering with Deduplication ID.	Serverless architectures, zero operational maintenance, automated cloud scaling.
Redis Streams	In-Memory Append Log	In-Memory with consumer group offsets & optional truncation (`MAXLEN`).	Strict ID ordering; Ultra-low <1ms latency.	Real-time chat, fast activity feeds, lightweight pub/sub with consumer groups.

3. Interactive Simulator: Point-to-Point vs Topic Fan-Out

Interactive Testbed

Experience the fundamental difference in message delivery: In Point-to-Point, messages balance across workers. In Pub/Sub Fan-Out, publishing an OrderPlaced event delivers a full copy to Payment, Inventory, and Notification services simultaneously!

Messaging Pattern:

Published Events: 0

Total Deliveries Processed: 0

Active Mode: 1-to-1 Load Leveling

💳 Payment ServiceIDLE

Processed: 0 messages

📦 Inventory ServiceIDLE

Processed: 0 messages

📧 Email NotificationIDLE

Processed: 0 messages

// Simulator ready. Switch modes and click "Publish 1 Event" to observe message routing...

4. Visualizing the Topic Fan-Out Architecture

Watch the live data flow: When the Order API publishes an OrderPlaced event to the Pub/Sub Exchange/Topic, it duplicates the event into three dedicated queues. Each microservice consumes at its own independent rate without blocking the others.

⚡ Pub/Sub Topic Fan-Out Topology

5. Message Ordering, Partition Keys & Delivery Semantics

How do distributed messaging systems scale horizontally while preserving strict message order for individual entities?

🔑 The Partition Key Mechanism (Kafka)Global ordering kills concurrency; Partition ordering scales infinitely.If you enforce a single global queue, throughput is throttled by a single consumer thread. Kafka solves this by partitioning topics into N partitions. By assigning a Partition Key (e.g. userId or accountId), all events for that specific user land on the exact same partition in strict sequential order!

🛡️ The 3 Delivery SemanticsTrade-offs in message delivery guarantees• At-Most-Once: Message delivered 0 or 1 time. High speed, acceptable loss (e.g. telemetry, metrics).
• At-Least-Once (Standard): Message delivered 1 or more times. Never loses data, but requires Idempotent Consumers.
• Effectively-Once: Achieved by pairing At-Least-Once delivery with transactional deduplication keys in DB.

02 / Knowledge Check

Real-World Scenario Quizzes

Test your understanding of messaging patterns, broker trade-offs, and partition ordering.

Q1: Your analytics team introduces a bug in their consumer worker that corrupted 3 days of calculated daily revenue metrics. If your architecture uses Apache Kafka, how do you recover?

Correct Answer: A! Unlike RabbitMQ or standard SQS (which delete messages upon acknowledgement), Kafka stores events as an immutable append-only commit log on disk for a configured retention period (e.g. 7 days). Resetting the consumer offset allows you to replay historical events at full speed.

Q2: A bank ledger system processes deposits and withdrawals across 20 concurrent worker servers. Why is it CRITICAL to set the Partition Key to account_id when publishing transaction events?

Correct Answer: B! If deposit and withdrawal events for the same account landed on random partitions, Worker 1 might process a withdrawal before Worker 2 processes the preceding deposit, causing an erroneous account overdraft. Hashing the account_id ensures strict per-account FIFO ordering.

Q3: An e-commerce system needs to notify 4 downstream services (Fraud, Payment, Inventory, Email) whenever an order is created. What happens if the Order Service invokes each of these 4 services via synchronous REST HTTP calls directly?

Correct Answer: B! Direct synchronous REST calls create tight temporal coupling. Using a Pub/Sub Topic Fan-Out decouples the Order Service: it publishes 1 event and returns immediately. If the Email service is temporarily down, its dedicated queue buffers messages until it recovers, with zero impact on user checkouts.

Q4: In a Point-to-Point Queue with 5 competing worker nodes, if Producer publishes 100 messages, how many times is each message processed in total?

Correct Answer: B! In Point-to-Point queueing (Competing Consumers pattern), messages are distributed across workers for load leveling. Each message is processed by exactly one worker.

03 / Practice Exercise

Design an Event-Driven Checkout Fan-Out Pipeline

Construct a decoupled Pub/Sub architecture: Connect the Order Service to a Central PubSub Topic, fan out to dedicated Subscriber Queues for Payment, Inventory, and Notifications, and connect each queue to its dedicated Consumer Service.

Interactive CanvasMode: Freeform Drag & Connect

Task: Event-Driven Pub/Sub Fan-Out

Grid Overlay

🎯

Scenario Prompt: Design a resilient e-commerce checkout fan-out pipeline: Connect the Order Service to the PubSub Topic. Fan out the PubSub Topic to 3 separate Subscriber Queues (Payment Queue, Inventory Queue, and Email Queue). Connect each queue to its respective Consumer Service (Payment Consumer, Inventory Consumer, Email Consumer), and wire Payment Consumer to the Database to record the transaction.

Available Components (Click to place on canvas):

🖱️Canvas is emptyClick components above to place them onto the canvas.
Drag components to arrange them freely, and click two nodes to connect them.

💡 Drag to move • Click Node A ➔ Node B to connect • Click ✕ to delete0 components placed