The Difference Between a Message Queue and an Event Stream

The Conflation Problem

"We need a queue" is how the conversation usually starts. Sometimes what the team actually needs is a queue — SQS, RabbitMQ. Sometimes they need an event stream — Kafka, Kinesis. These systems have overlapping use cases at the surface and divergent behavior underneath. The conflation causes real problems: teams build on SQS and discover they need replay capability; teams adopt Kafka and discover they have taken on operational complexity for a use case that SQS would have handled more simply.

The core difference is in consumer semantics and retention model.

Message Queues: Consume Once, Then Delete

A message queue (SQS, RabbitMQ) delivers a message to one consumer, waits for acknowledgment, and deletes the message. The message is consumed exactly once (with at-least-once delivery guarantees — if acknowledgment fails, the message is redelivered). The queue holds messages only until they are consumed or a retention period expires.

This model is appropriate for task distribution: background jobs, work that needs to be done exactly once by one worker. Sending an email, resizing an image, processing a payment. Each task needs to happen once. Multiple workers consume from the same queue, each getting a different task. The queue distributes work.

What a message queue does not give you: the ability to replay messages, multiple independent consumers for the same message, or a persistent log of everything that was processed.

Event Streams: Persistent Log, Multiple Consumers

An event stream (Kafka, Kinesis) is a durable, ordered log of events. Producers append events to the log. Consumers read from the log at their own offset — each consumer group maintains an independent position in the log. Events are not deleted when consumed; they are retained for a configured period (hours, days, weeks) or forever (with Kafka's log compaction).

This model is appropriate when: multiple independent services need to react to the same event, you need the ability to replay events (for new consumers, for backfill, for debugging), or you need a persistent audit trail of state changes.

# Message Queue semantics (SQS):
Producer -> [message] -> Queue -> Consumer A (message deleted after ack)
                               -/ Consumer B (never receives the message)

# Event Stream semantics (Kafka):
Producer -> [event] -> Topic/Partition -> Consumer Group A (offset advances)
                                       -> Consumer Group B (independent offset)
                                       -> Consumer Group C (can replay from beginning)
# Event persists in the log until retention expires

The Operational Difference

SQS is fully managed. You send messages, they arrive at your consumer. No brokers to run, no partition management, no broker replication to configure. Throughput scales automatically. The operational overhead is minimal.

Kafka requires running brokers (or using a managed service like Confluent Cloud or MSK). Partition count affects parallelism and must be planned. Consumer group offsets must be managed. Log retention and compaction must be configured. Storage grows with retention period and write volume. The operational overhead is significant — a Kafka cluster is a system to operate, not just a service to call.

This is not an argument against Kafka. It is an argument that Kafka's operational overhead should be justified by the use case.

Choosing Between Them

Use a message queue (SQS, RabbitMQ) when:

• Each message needs to be processed by exactly one worker
• You do not need replay capability
• The workload is task distribution (background jobs)
• Operational simplicity is a priority

Use an event stream (Kafka, Kinesis) when:

• Multiple independent consumers need to process the same event
• You need replay capability (new consumers, backfill, debugging)
• You need an ordered, persistent log of events
• Event sourcing or CQRS patterns are in use

A common hybrid: use SQS for task distribution (one worker per task), and use Kafka for domain events that multiple services need to react to. They serve different purposes. Run both if you have both use cases. Do not use Kafka for everything because it is more powerful — the power comes with a cost.