Message Queues and Event-Driven Architecture: RabbitMQ, Kafka, and Redis Pub/Sub

Event-driven architecture decouples services through asynchronous message passing, improving resilience and scalability. But the message broker you choose determines the trade-offs you live with. RabbitMQ, Apache Kafka, and Redis Pub/Sub each have fundamentally different delivery guarantees, throughput characteristics, and operational profiles.

RabbitMQ: Smart Broker, Durable Queues

RabbitMQ is the most mature message broker, designed around the AMQP protocol. It excels at complex routing patterns—direct exchanges, topic exchanges, headers exchanges—and guarantees message delivery with publisher confirms and consumer acknowledgments:

import pika

# Publisher
connection = pika.BlockingConnection(
    pika.ConnectionParameters('rabbitmq.example.com'))
channel = connection.channel()

channel.exchange_declare(exchange='orders', exchange_type='topic')

message = {
    'order_id': 'ORD-123',
    'user_id': 'USR-456',
    'total': 2999,
    'items': [
        {'product_id': 'PROD-789', 'quantity': 2},
    ],
}

channel.basic_publish(
    exchange='orders',
    routing_key='order.created',
    body=json.dumps(message),
    properties=pika.BasicProperties(
        delivery_mode=2,  # Persistent
        content_type='application/json',
    ),
)

# Consumer
def callback(ch, method, properties, body):
    event = json.loads(body)
    print(f"Processing order {event['order_id']}")
    # Process the order...
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(
    queue='order_notifications',
    on_message_callback=callback,
    auto_ack=False,  # Manual acknowledgment
)
channel.start_consuming()

RabbitMQ shines for workloads requiring complex routing, per-message acknowledgments, and dead-lettering for failed messages. It's ideal for transactional workflows where each message must be processed exactly once.

The trade-off: RabbitMQ stores messages on disk only when queues grow beyond memory limits. Under sustained high throughput, it becomes disk-bound faster than Kafka. It handles tens of thousands of messages per second comfortably; beyond that, Kafka takes over.

Apache Kafka: Immutable Log, High Throughput

Kafka treats messages as an immutable, ordered commit log. Producers append events to a topic; consumers maintain their position (offset) in the log. This model enables replay—consumers can rewind and reprocess events:

from kafka import KafkaProducer, KafkaConsumer
import json

# Producer
producer = KafkaProducer(
    bootstrap_servers=['kafka-1:9092', 'kafka-2:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    acks='all',          # Wait for all replicas to acknowledge
    compression_type='gzip',
)

producer.send('orders', {
    'event_type': 'order.created',
    'order_id': 'ORD-123',
    'timestamp': '2026-06-23T12:00:00Z',
})

# Consumer
consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['kafka-1:9092', 'kafka-2:9092'],
    group_id='notification-service',
    auto_offset_reset='earliest',
    enable_auto_commit=False,
)

for msg in consumer:
    event = json.loads(msg.value)
    print(f"Offset {msg.offset}: {event['order_id']}")
    consumer.commit()

Kafka's throughput advantage comes from sequential disk I/O, batching, and zero-copy data transfer. A well-tuned cluster handles millions of messages per second. Message retention is time-based (default 7 days) or size-based, giving consumers a window for reprocessing.

Kafka is the best choice for event sourcing, audit logging, stream processing, and any use case where message ordering and replay are critical. The trade-off: higher operational complexity. Kafka requires ZooKeeper (or KRaft in newer versions), careful partition sizing, and monitoring of consumer lag.

Redis Pub/Sub: Lightweight and Fast

Redis Pub/Sub is the simplest of the three—publishers send messages to channels, subscribers receive them in real time. There's no message persistence, no acknowledgment mechanism, and no replay:

import redis

r = redis.Redis(host='redis.example.com', port=6379)

# Publisher
r.publish('order:notifications', json.dumps({
    'order_id': 'ORD-123',
    'event': 'status_change',
    'new_status': 'shipped',
}))

# Subscriber
pubsub = r.pubsub()
pubsub.subscribe('order:notifications')

for message in pubsub.listen():
    if message['type'] == 'message':
        event = json.loads(message['data'])
        print(f"Order {event['order_id']} is now {event['new_status']}")

Redis Pub/Sub is perfect for real-time notifications, live dashboards, chat messages, and any use case where lost messages during subscriber downtime are acceptable. It's the simplest to deploy and operate.

The critical limitation: if a subscriber disconnects, it misses all messages published during the disconnect. There's no queue, no replay. For "fire and forget" real-time broadcasts, Redis Pub/Sub is ideal. For anything requiring delivery guarantees, choose RabbitMQ or Kafka.

Comparison Matrix

| Feature | RabbitMQ | Kafka | Redis Pub/Sub | |---|---|---|---| | Delivery guarantees | At-least-once, exactly-once (optional) | At-least-once, exactly-once (Kafka Streams) | At-most-once | | Message persistence | Disk when queues grow | All messages to disk | None | | Throughput | 10K–50K msg/s per node | 100K–1M+ msg/s per node | 100K+ msg/s | | Message ordering | Per-queue ordering | Per-partition ordering | Per-channel ordering | | Consumer groups | Native | Native | Not supported | | Replay capability | Limited (queue re-publish) | Native (offset reset) | None | | Operational complexity | Moderate | High | Low | | Routing flexibility | Excellent (exchanges) | Limited (topic string) | Fixed (channels) |

Hybrid Architectures

Many production systems use multiple brokers for different jobs. A typical pattern: Kafka for the event log and stream processing, RabbitMQ for work queues with complex routing, and Redis Pub/Sub for real-time browser notifications via WebSocket.

Events flow from services into Kafka as the source of truth. RabbitMQ consumers read from Kafka topics and dispatch work to queues with specific routing. Redis Pub/Sub broadcasts real-time updates to connected WebSocket clients.

Build Event-Driven Systems with SoniNow

Message queues and event-driven architecture decouple your services, enabling independent scaling and fault isolation. SoniNow architects event-driven systems that choose the right messaging backbone for each communication pattern in your application.