RabbitMQ vs Kafka: Message Broker vs Event Log

Almost every integration project I advise on eventually reaches the same fork in the road. Two systems need to talk asynchronously, the team has decided that direct point-to-point calls will not scale, and someone in the room says the word "queue" while someone else says the word "Kafka". Within ten minutes the conversation has turned into a contest, as if RabbitMQ and Apache Kafka were two brands of the same thing and the job was to declare a winner. They are not the same thing. Comparing them fairly means understanding that one is a message broker and the other is a distributed log, and that the difference between a queue and a log changes almost everything downstream. This guide walks through that difference in plain terms, stays fair to both, and gives you a way to choose that survives contact with real requirements. If you want the wider context first, this piece sits underneath my enterprise system integration pillar, which explains where asynchronous messaging fits in the overall integration landscape.

The message up front: RabbitMQ excels at getting a unit of work to the right consumer, once, with rich routing and low latency, then discarding it. Kafka excels at recording a stream of events durably so that many independent consumers can read and re-read it at their own pace. If your mental model is "distribute tasks to workers", think RabbitMQ. If your mental model is "publish a stream of facts that many systems consume", think Kafka. Most of the confusion comes from forcing one shape onto the other's problem.

1. Two different tools for two different jobs

The single most useful thing you can do before comparing RabbitMQ and Kafka is to stop treating them as competitors and start treating them as answers to different questions. RabbitMQ answers "how do I hand this piece of work to whichever worker is free, reliably, and know it got done?" Kafka answers "how do I keep an ordered, durable record of everything that happened so any number of systems can consume it now or later?" Those are not the same question, and the tool that is superb at one is only adequate at the other.

This is why benchmark wars miss the point. You can find a benchmark where Kafka moves millions of messages per second and a benchmark where RabbitMQ delivers a message in under a millisecond, and both are true, and neither tells you which fits your problem. The right frame is not speed, it is shape. Is your workload a set of tasks to be distributed and completed, or a stream of events to be recorded and replayed? Get the shape right and the choice usually makes itself. Get it wrong and you spend the next year fighting the tool. For a foundational look at the task-distribution pattern itself, see my companion piece on what a message queue is, and for the streaming side, the introduction to event streaming.

2. RabbitMQ: a message broker

RabbitMQ is a mature, general-purpose message broker. Its native protocol is AMQP (the Advanced Message Queuing Protocol), and it also speaks MQTT, STOMP and others through plugins. Its job is to accept messages from producers, decide where they should go, and deliver them to consumers, then remove them once they have been acknowledged. The word that matters is "broker": RabbitMQ sits in the middle and actively routes.

The routing is the part people underrate. In AMQP, a producer does not publish directly to a queue. It publishes to an exchange, and the exchange decides which queues receive a copy based on rules called bindings. A direct exchange routes by an exact routing key. A topic exchange routes by pattern matching on the key, so a message keyed order.eu.priority can fan out to whichever queues subscribed to order.eu.* or order.#. A fanout exchange copies to every bound queue. A headers exchange routes on message attributes. This gives you expressive, per-message control over delivery that is genuinely hard to replicate elsewhere without writing code.

Once a message lands in a queue, RabbitMQ pushes it to a consumer. The consumer processes it and sends an acknowledgement. On acknowledgement, the broker deletes the message; the work is done and there is no reason to keep it. If the consumer dies before acknowledging, the message is requeued and delivered again, which is what makes the broker reliable for task processing. You can spread load across many consumers on the same queue, and RabbitMQ hands each message to exactly one of them, giving you competing-consumer work distribution out of the box. Add features like per-message priority, time-to-live, dead-letter exchanges for messages that repeatedly fail, and publisher confirms, and you have a rich toolkit for orchestrating asynchronous work. RabbitMQ is, in short, a smart post office: it takes your letter, works out who should get it, delivers it, and once the recipient signs for it the letter is gone.

3. Kafka: a distributed log

Apache Kafka started life at LinkedIn and is built on a fundamentally different idea. It is not a broker that routes and discards. It is a distributed, append-only commit log. Producers append records to the end of a topic, and a topic is physically split into partitions, each of which is an ordered, immutable sequence of records. Every record in a partition gets a monotonically increasing number called its offset. Kafka does not delete a record when someone reads it. The record stays on disk for as long as the retention policy allows, whether that is seven days, thirty days, or forever.

Because the log is retained rather than consumed away, reading works completely differently. A consumer does not wait to be pushed a message and then acknowledge its deletion. Instead, the consumer pulls records and tracks its own position, its offset, in each partition. Consumer A can be reading offset 1,000,000 while consumer B, doing something entirely different with the same data, is still at offset 50,000, and neither affects the other. If you add a brand new consumer next year, it can start from offset zero and replay the entire history. This is the defining property of the log model: the data is a durable record that many independent readers consume at their own pace, not a pile of work that vanishes as it is handled.

Ordering in Kafka is guaranteed within a partition, not across a whole topic. Records that must stay in order need to share a partition, which you arrange by giving them the same partition key (for example, keying by customer ID so all of one customer's events land in one partition and stay ordered). Consumers scale through consumer groups: Kafka assigns each partition to exactly one consumer within a group, so parallelism is bounded by partition count, and each group tracks its own offsets independently. The result is a platform built for high-throughput, durable, replayable event streams feeding many consumers. Kafka is not a post office. It is more like a ship's logbook that everyone can read, that never erases an entry, and where each reader keeps their own bookmark. For a deeper look at Kafka's internals on their own terms, see my Kafka explained piece.

4. The core difference: queue versus log

Everything else follows from one structural choice. In a queue, a message is delivered and then removed; it exists to be consumed. In a log, a record is appended and retained; it exists to be read, re-read and replayed. That difference in what happens to the data after it is read is the whole comparison in miniature. The diagram below shows both models side by side: on the left, RabbitMQ routing messages into queues from which consumers take work that is then gone; on the right, Kafka appending records to a log that consumers read at their own offsets while the records stay put.

Read the two halves and the trade-offs become obvious. On the left, the queue gives you routing intelligence and clean once-and-done delivery, but once a message is acknowledged it is gone; there is nothing to replay and no shared history. On the right, the log gives you durability, replay and many independent readers, but there is no per-message routing and no automatic deletion on read; the consumer, not the broker, is responsible for tracking position. Neither is better in the abstract. They are optimised for opposite ends of the same spectrum.

5. Head to head

With the structural difference clear, here is the direct comparison across the dimensions that actually drive architecture decisions. Read it as a map of trade-offs rather than a scorecard; nearly every row is a case of "different by design", not "one wins".

A caution on the throughput row: do not choose Kafka simply because a headline says it is faster. Its throughput advantage shows up with large, sustained, sequential streams and enough partitions to parallelise. For modest volumes of individually routed tasks, RabbitMQ is typically lower-latency and far simpler to run. "Kafka is faster" is a statement about a specific workload shape, not a universal truth, and picking the wrong tool because of a benchmark you will never reproduce is one of the more common and expensive mistakes I see.

6. Where RabbitMQ fits

RabbitMQ is the right answer when your problem is distributing discrete units of work and you need each unit handled once by whichever worker is available. Think of a web application that offloads slow jobs: image resizing, PDF generation, sending notification emails, running a report. The application publishes a task, a pool of workers competes for it, one worker processes it and acknowledges, and the task is done and forgotten. This competing-consumer pattern is RabbitMQ's home ground, and it does it with very little ceremony.

It also shines when routing logic is genuinely complex. If different messages must reach different consumers based on content, region, priority or type, the exchange-and-binding model lets you express that declaratively instead of writing dispatch code. A topic exchange routing on keys like invoice.eu.overdue versus invoice.us.new gives you fine-grained fan-out that would be awkward to reproduce with a partitioned log. Add priority queues for urgent work, per-message TTL, delayed delivery, and dead-letter exchanges that catch messages which fail repeatedly so a human can inspect them, and RabbitMQ becomes a very capable orchestration layer for asynchronous work.

In enterprise integration specifically, RabbitMQ is often the quiet workhorse between systems that need reliable, decoupled request-and-response or command-style messaging without the operational weight of a streaming platform. When I connect an ERP such as Microsoft Dynamics 365 Business Central to downstream services and the pattern is "here is a command, process it once, tell me it is done", a broker fits more naturally than a log. If that ERP integration scenario is your world, my write-up on Business Central APIs and integrations covers where a broker slots into that architecture. The rule of thumb: if you would describe the payloads as "jobs" or "commands" that are done once and discarded, RabbitMQ is very likely your tool.

7. Where Kafka fits

Kafka is the right answer when your data is a stream of events that multiple systems need to consume, possibly at different times and different speeds, and when the historical record itself has value. The classic example is a stream of business events, orders placed, payments taken, inventory moved, shipments dispatched, published once to a topic and consumed independently by the fulfilment system, the analytics warehouse, the fraud detection service and the customer notification service. Each consumer reads at its own pace, none interferes with the others, and a new consumer added later can replay the whole history to catch up. A queue cannot do this cleanly because the first consumer to read would remove the message.

Replay is often the deciding factor. Because Kafka retains records, you can reprocess history: fix a bug in a consumer and re-run it over the last thirty days of events, backfill a new database from the log, or bootstrap a fresh microservice from the full event history. This makes Kafka a natural backbone for event-driven architectures and for the event-sourcing pattern, where the log of events is treated as the authoritative source of truth and current state is derived from it. It is also the foundation for stream processing, where frameworks continuously transform, join and aggregate these streams in near real time.

The other place Kafka dominates is high-volume ingestion: log and metric aggregation, clickstream capture, IoT telemetry, any firehose where you need to absorb an enormous sequential flow durably and feed it to many downstream consumers. Its partitioned, sequential-write design is built precisely for that. If you would describe your payloads as "events" or "facts" that many systems care about and that you may want to replay, Kafka is very likely your tool. For the broader pattern behind this, my event streaming introduction sets out why the durable-stream model has become so central to modern integration.

8. How to choose

Strip away the tribalism and the choice comes down to a few honest questions about the shape of your problem. Ask them in order and the answer usually settles itself.

• Is the payload a task or an event? A task is something to be done once by one worker (resize this image, send this email). An event is a fact that happened and that several systems may care about (an order was placed). Tasks lean RabbitMQ, events lean Kafka.
• Do multiple independent consumers need the same data? If one and only one worker should handle each message, a queue fits. If several unrelated systems each need their own full copy of the stream, a log fits.
• Do you need to replay history? If reprocessing past messages, backfilling new services, or auditing a durable record matters, that is a strong pull toward Kafka. If a message is worthless once handled, RabbitMQ's discard-on-ack is a feature, not a limitation.
• How complex is the routing? Rich, per-message, content-based routing is RabbitMQ's strength. Kafka expects consumers to filter for themselves; it routes by partition, not by rule.
• What is the volume and shape? Enormous, sustained, sequential streams favour Kafka's design. Modest volumes of individually addressed work favour RabbitMQ's simplicity and low latency.
• What can your team actually operate? Kafka is a more involved platform to run well, with partitions, consumer-group rebalancing and retention to manage. RabbitMQ is generally simpler to stand up for classic messaging. Operational reality is a legitimate input, not a cop-out.

The insight that resolves most debates: many mature systems run both, because they have both shapes of problem. Kafka carries the durable event backbone that feeds analytics, stream processing and multiple services, while RabbitMQ handles command-style task distribution and complex routing between specific systems. Choosing "RabbitMQ or Kafka" for the whole organisation is often the wrong question. Choosing the right one per workload, and being willing to use both, is how the strongest architectures I have worked on are actually built. If you want the map of how these pieces fit together, start from the enterprise system integration pillar.

9. References

The claims in this guide track the primary documentation and specifications for both platforms. Rather than link deep pages that move over time, go to the authoritative sources directly:

• Apache Kafka documentation, the official docs covering the log model, topics, partitions, offsets, consumer groups, retention and delivery semantics.
• RabbitMQ documentation, the official docs covering exchanges, bindings, queues, acknowledgements, routing types and delivery guarantees.
• AMQP (Advanced Message Queuing Protocol), the open protocol specification underpinning RabbitMQ's core messaging model.

Final thoughts

RabbitMQ and Kafka are both excellent, and the endless "which is better" framing does a disservice to both. RabbitMQ is a message broker: it routes work intelligently, delivers it once, and clears it away when the job is done. Kafka is a distributed log: it records events durably so that many consumers can read and replay them at their own pace. The queue-versus-log distinction is not a detail, it is the whole thing, and once you internalise it the right tool for a given workload stops being a matter of opinion and becomes a matter of fit.

My advice after two decades of connecting enterprise systems is to resist picking a favourite and instead learn to read the shape of each problem. Tasks to distribute, once and done, with rich routing? Reach for RabbitMQ. A durable stream of events that many systems consume and may need to replay? Reach for Kafka. And when a real architecture has both kinds of problem, which most do, use both without apology. The mark of a good integration engineer is not loyalty to a tool. It is matching the tool to the job, honestly, every time.

Related reading: Enterprise system integration explained, What is a message queue, Kafka explained, What is event streaming, Business Central APIs and integrations.