Kafka Explained for Business Users

If you spend any time near a modern data platform, you will hear the word Kafka within the first hour. It is quietly running underneath a huge amount of the real-time software that large organisations depend on, moving events between systems at a scale and speed that traditional databases and message queues were never built to handle. And yet, for something so central, Kafka is remarkably badly explained. The typical introduction opens with distributed commit logs, partitions, offsets and consumer group rebalancing, and a business reader closes the tab before the second paragraph. This guide takes the opposite approach. It explains what Kafka actually is, in language a decision maker can use, and it does so as one piece of the wider integration picture rather than as an isolated piece of infrastructure trivia.

Start with the bigger picture: Kafka is one tool within the broader discipline of connecting systems together. If you have not already, read the enterprise system integration pillar first. It frames why organisations need to move data between systems at all, and Kafka makes far more sense once you understand the problem it is solving rather than just the mechanism it uses.

1. What Kafka is, in one minute

Apache Kafka is an open-source system for moving streams of data between the applications that produce that data and the applications that need to consume it. It was originally built at LinkedIn to handle the enormous volume of activity data the site generated, and it was donated to the Apache Software Foundation, where it became one of the most widely used pieces of data infrastructure in the world. Today it underpins real-time systems at a large share of major enterprises, from banks and retailers to logistics and telecommunications companies.

The one-sentence version is this: Kafka lets one part of your business publish a continuous stream of events, and lets any number of other parts of the business read that same stream, independently, in real time, without any of them needing to know about each other. A point-of-sale system publishes a stream of sales. A fraud detection system reads it. A stock-replenishment system reads it. A dashboard reads it. None of them talk to each other directly, and adding a fifth consumer tomorrow does not disturb the four already there. That decoupling, at very high volume and with strong durability guarantees, is the whole value.

People describe Kafka in several ways depending on their background. Developers call it a distributed event streaming platform. Data engineers call it a durable message bus. Architects call it the central nervous system of the enterprise. All three are pointing at the same thing from different angles. Underneath every one of those descriptions is a single, surprisingly simple idea, and once you understand that idea the rest of Kafka falls into place quickly.

2. The big idea: an append-only log

The concept at the heart of Kafka is the append-only log. Forget databases, queues and messaging for a moment and picture a notebook in which you only ever write new lines at the bottom. You never erase a line, you never insert between lines, and you never rewrite an old line. You only add to the end. Each new line gets the next number in sequence, so line 1, then line 2, then line 3, and so on forever. That is an append-only log, and it is the single structure on which all of Kafka is built.

This sounds almost too simple to be powerful, but the simplicity is exactly where the power comes from. Because you only ever append, writing is extremely fast; the system never has to hunt for the right place to insert or update. Because every entry has a fixed sequence number that never changes, any reader can remember exactly how far they have read and pick up precisely where they left off. And because nothing is ever deleted on read, ten different readers can each read the same log at their own pace without interfering with one another. A record being read by one consumer is not consumed or removed; it just sits in the log, available to the next reader.

This is the deepest difference between Kafka and a traditional message queue. In a classic queue, a message is delivered to a consumer and then removed, gone once handled. In Kafka, the log is the source of truth and reading is a non-destructive act. The record stays; only your position in the log moves forward. That one design decision is what makes replay, multiple independent consumers and long retention possible, and we will come back to each of those. For now, hold on to the picture of a notebook you only ever add lines to, where every reader keeps their own bookmark.

3. Topics and partitions

In Kafka, that append-only log is called a topic. A topic is a named stream of records about one kind of thing: a topic for orders, a topic for temperature readings, a topic for user logins. Producers write records to a topic, and consumers read records from it. If you want to move a new kind of data through Kafka, you create a new topic for it, and the naming usually reads like a description of the event, such as orders, payments or sensor-readings.

A single log on a single machine would eventually hit a ceiling on how much it could handle, so Kafka splits each topic into partitions. A partition is one slice of the topic, itself an append-only log, and the partitions of a topic can live on different machines. Splitting a topic into partitions is how Kafka scales: instead of one log doing all the work, you have several logs sharing the load, and you can add more as volume grows. Within any single partition, order is strictly guaranteed, so record 5 always comes after record 4 in that partition, but across partitions there is no global ordering, which is a trade-off worth understanding when order matters to your business.

Each record inside a partition sits at a numbered position called an offset, the sequence number from our notebook analogy. Offset 0 is the first record ever written to that partition, offset 1 the next, and so on. A consumer reading a partition simply tracks the offset it has reached. The diagram below shows a single topic split into partitions, producers appending new records to the end of each partition, and two separate consumer groups each reading at their own independent position.

The important thing to take from the diagram is that the log is written once and read many times. Producers only ever add to the right-hand end. Each consumer group keeps its own bookmark and moves through the records at whatever speed suits it. Billing can be an hour behind analytics and neither one is affected by the other. That is the shape of nearly every Kafka deployment, no matter how large.

4. Producers, consumers and consumer groups

A producer is any application that writes records into a topic. It could be a website recording clicks, a payment gateway recording transactions, or an industrial gateway recording sensor readings. The producer does not know or care who will read the data. It simply appends records to a topic and moves on. This is a genuine liberation for the system that produces the data, because it never has to be changed when a new downstream system wants a copy of the stream.

A consumer is any application that reads records from a topic. It subscribes to a topic and processes records in order, tracking the offset it has reached so it can resume from the right place after a restart. Because reading does not delete anything, a consumer can be stopped, upgraded and restarted, and it simply carries on from its last recorded offset. If it needs to reprocess, it can move its bookmark backward and read older records again, which is a capability traditional queues rarely offer.

The idea that ties this together and confuses newcomers most is the consumer group. A consumer group is a set of consumer instances that cooperate to read a topic as a single logical unit, dividing the partitions between themselves so that each partition is handled by exactly one member of the group at a time. If a topic has six partitions and a group has three consumers, each consumer handles two partitions, and the work is shared. Add a fourth consumer and Kafka rebalances the partitions across the four automatically. Lose a consumer and the survivors pick up its partitions. This is how Kafka scales the reading side to keep up with high-volume topics.

The second half of the consumer group idea is what makes Kafka a broadcast system as well as a work-sharing one. Different consumer groups are completely independent of one another. The analytics group and the billing group each read the full stream and each keep their own offsets. Within a group, the partitions are shared for scale; across groups, the whole stream is duplicated for independence. That combination, load sharing inside a group and full independence between groups, is exactly what lets a single orders topic feed a fraud system, a warehouse system, a reporting system and a customer notification system all at once, each at its own pace, none aware of the others. This is the publish and subscribe pattern operating at scale, and it is worth reading the publish and subscribe explainer alongside this section to see how the general pattern maps onto Kafka specifically.

5. Retention and the power of replay

Because Kafka never deletes a record just because it has been read, a natural question follows: when does anything get removed at all? The answer is retention. You configure how long Kafka keeps records in a topic, and it keeps them for that period regardless of how many times they have been read. Retention can be set by time, for example seven days or thirty days, or by size, for example the last five hundred gigabytes, and only when a record ages past the retention limit is it finally discarded. Within the retention window, every record is available to any consumer that wants it.

This turns Kafka into something more interesting than a pipe. Within the retention window, the topic is a durable record of everything that happened, in order, that any system can read from the beginning or from any point in the middle. That capability is called replay, and it is one of Kafka's most valuable and least appreciated features for a business audience.

Why replay matters in practice: suppose you build a new reporting system and want it populated with the last two weeks of orders. With Kafka, you point it at the orders topic, set its bookmark to the start of the retention window, and it reads the entire history to catch up, then carries on live. Suppose a consumer had a bug and processed a day of data incorrectly. You fix the code, move the bookmark back to before the bad day, and reprocess. No special export, no database restore, no begging another team for a data extract. The stream is the record, and you can read it again.

This is why event streaming is treated as an architectural style and not just a transport mechanism. When the log of events is durable and replayable, it becomes a shared source of truth that new systems can be built against at any time, including systems that did not exist when the events were first produced. For the wider treatment of this idea, the event streaming explainer covers how organisations design around durable event logs rather than around request and response calls.

6. Core concepts glossary

Almost every Kafka conversation is built from the same small vocabulary. Once you can attach a plain-English meaning to each of the six core terms below, most Kafka documentation and most vendor discussions become readable. Keep this table handy as a reference.

Two of these terms deserve a note. A broker is simply a Kafka server; a production Kafka cluster is a group of brokers working together, each holding some partitions and keeping backup copies of others so that if one broker fails, another already has the data and no records are lost. That replication across brokers is a core reason Kafka is trusted with important data. And the pairing of partition and offset is what makes the whole model work: the partition says which log, the offset says which line, and together they pinpoint any record exactly.

7. What Kafka is great at and what it is not for

Kafka is exceptional at a specific set of problems, and understanding where it shines keeps you from forcing it into places it does not belong. It is outstanding at moving high volumes of events continuously, at fanning a single stream out to many independent consumers, at retaining a durable and replayable history of events, and at decoupling the systems that produce data from the systems that use it so each can evolve on its own schedule. When you have a firehose of events that several systems all need, in order, reliably, at scale, Kafka is very hard to beat, and that is precisely the situation it was designed for.

It is also worth being clear about what Kafka is not. It is not a database you run rich queries against; it stores an ordered log, not tables you can search by arbitrary fields. It is not a simple task queue for a small application, where a lightweight message broker would be easier to run and entirely sufficient. It is not the right tool for classic request and reply interactions where one system asks another a question and waits for a single answer; that is an API's job, not a stream's. And it is genuinely heavier to operate than a small queue, so reaching for it on a low-volume workload usually means paying an operational cost you did not need to.

The honest caution: Kafka is powerful, and that power tempts teams to make it the answer to every integration question. It is not. Running a Kafka cluster well takes real operational skill, and a great many integration problems are better solved with a simple API call or a modest message queue. Choose Kafka when you genuinely have a high-volume stream that multiple systems need to consume independently, with a durable replayable history. If your requirement is one system asking another a single question, an API is simpler, cheaper and easier to reason about.

The decision between Kafka and a traditional broker comes up so often that it deserves its own treatment. If you are weighing the two, the RabbitMQ versus Kafka comparison works through the trade-offs directly: RabbitMQ excels at flexible routing and per-message work queues, while Kafka excels at high-throughput durable streams read by many consumers. They are different tools for different shapes of problem, and knowing which shape you have is most of the decision.

8. Where Kafka fits in enterprise integration

Zoom out from the mechanics and Kafka is best understood as one option within the broader discipline of enterprise integration, the work of getting an organisation's systems to share data reliably. For decades that sharing was done mostly through direct connections and scheduled batch files: system A calls system B, or system A drops a file overnight that system B picks up in the morning. Those patterns still have their place, but they struggle when data needs to move continuously, in real time, to many destinations at once. That is the gap event streaming platforms like Kafka fill.

In a well-designed estate, Kafka often becomes the central spine along which events flow. Instead of building a fragile web of point-to-point connections, where every new system has to be wired individually to every other system it needs, you publish events to Kafka once and let any system subscribe. The order event is produced a single time and consumed by finance, logistics, analytics and customer communications independently. When a new system arrives, it subscribes to the streams it needs without anyone having to modify the producers. That shift, from a tangle of direct links to a shared stream of events, is one of the most consequential architectural moves a large organisation can make, and it is exactly the transition the enterprise integration pillar describes in full.

Kafka rarely stands alone, though. It usually lives alongside the request and response APIs that business systems expose, and a mature integration approach uses each for what it does best. Streams carry the continuous flow of events; APIs handle the direct, synchronous questions one system asks another. A finance platform such as Microsoft Dynamics 365 Business Central, for instance, is integrated primarily through its APIs for reading and writing records on demand, and those API calls can sit comfortably beside a Kafka stream that broadcasts the resulting events onward. The Business Central APIs and integrations guide shows the API side of that same picture, and the two together, streams and APIs, are how real enterprise data moves. Choosing correctly between them, case by case, is a large part of what enterprise integration work actually involves, and it is the judgement that separates a clean, durable data platform from an expensive one that fights itself.

9. References

Two primary sources are worth knowing if you want to go deeper than this explainer. The first is the Apache Kafka documentation, the official documentation maintained by the Apache Software Foundation, which is the authoritative and continually updated reference for how Kafka works, from core concepts through to operational detail. The second is the original Kafka design paper, "Kafka: a Distributed Messaging System for Log Processing", published by the team at LinkedIn who built it, which lays out the reasoning behind the append-only log design and remains a clear window into why Kafka is shaped the way it is. Between the current documentation and the founding design paper, you have both the present-day reference and the original intent, and everything in this article traces back to those two sources.

Final thoughts

For all its reputation as a heavyweight piece of infrastructure, Kafka rests on an idea a non-technical reader can hold in their head: a notebook you only ever add lines to, where every reader keeps their own bookmark and nothing is erased when it is read. Topics are those notebooks, partitions are the slices that let them scale across machines, offsets are the line numbers, producers add the lines, consumer groups share the reading and stay independent of one another, and retention decides how long the lines are kept. That is the whole model, and it is genuinely most of what a business decision maker needs to understand.

The reason Kafka matters is not the cleverness of its internals but what that simple model enables: real-time streams of events that many systems can consume independently, a durable and replayable record of what happened, and a clean separation between the systems that produce data and the systems that use it. Those properties are exactly what a modern, event-driven organisation needs, which is why Kafka has become the backbone of so much real-time data. Use it where you truly have that shape of problem, keep simpler tools for simpler needs, and treat it as one instrument within the wider integration picture rather than the answer to every question. Do that, and Kafka stops being an intimidating black box and becomes what it actually is: a well-designed way to move the events your business runs on.

Related reading: Enterprise system integration explained, What is event streaming, RabbitMQ vs Kafka, What is publish and subscribe, Business Central APIs and integrations.