What Is Event Streaming?

Most enterprise systems are built around state. A record sits in a table, you update it, and the new value overwrites the old one. That model is simple and it has served software well for decades, but it quietly throws away something valuable: the story of how the record got to its current value. Event streaming flips the emphasis. It says that the sequence of changes, the events themselves, is the primary thing worth keeping, and that current state is just something you can rebuild from that sequence whenever you need it. If you want the wider picture of how systems exchange data across an organisation, start with the pillar on enterprise system integration, then come back here for the streaming-specific view.

The idea up front: an event is an immutable statement that something happened at a point in time. Event streaming is the practice of capturing those statements as they occur, storing them in order, and letting any number of systems read and react to them independently. Once you see your business as a stream of events rather than a pile of current values, a lot of hard integration problems become easier.

1. What event streaming is

Event streaming is a way of moving and processing data as a continuous, ordered sequence of events rather than as periodic bulk transfers or one-off request-and-response calls. An event is a small, self-contained fact: "order 4471 was placed", "sensor 12 reported 74 degrees", "customer 89 changed their address". Each event is produced by a source system the moment the thing happens, written to a durable log, and made available for other systems to consume at their own pace.

The mental shift is the important part. In a traditional application you might run a nightly job that reads the orders table and pushes a summary to the warehouse. In an event-streaming architecture the order service emits an "order placed" event the instant the order is confirmed, and the warehouse, the analytics platform, the fraud checker and the loyalty system all read that same event and do their own work with it. Nobody polls a database. Nobody waits for a batch window. The data flows continuously, and each consumer is decoupled from the others and from the producer.

Three properties distinguish a stream from an ordinary message pipe. First, events are ordered, at least within a partition, so consumers can reason about sequence. Second, events are retained for a configurable period rather than deleted the moment they are read, which means a new consumer can join later and replay history. Third, many independent consumers can read the same stream without interfering with one another. Those three properties, order, retention and independent replayable consumption, are what make streaming qualitatively different from a plain queue, and they are the reason it underpins so many modern data architectures.

2. Events as a source of truth

The most powerful idea in event streaming is that the log of events can be the authoritative record, not a side effect of it. This is the heart of event sourcing, a pattern that has been discussed in the software design literature for years. In a conventional system, when a bank account balance changes, you update the balance field and the old value is gone. In an event-sourced system, you never store the balance directly. You store every deposit and withdrawal as an event, and the balance is computed by replaying those events in order. The event log is the truth; the balance is a derived, disposable view.

That inversion has real consequences. Because every change is retained as an immutable fact, you get a complete and trustworthy audit trail for free. You can answer not only "what is the balance now" but "what was the balance last Tuesday at noon", simply by replaying events up to that point. You can build a brand new read model, a new report, a new search index, a new cache, by replaying the log into it, without touching the source system. And when a downstream consumer has a bug, you fix the code and reprocess the history rather than begging the source for a data extract that may no longer exist.

The trade-off is honesty about complexity. Treating events as the source of truth means you cannot simply "edit a row" to correct a mistake; you record a compensating event instead, because the log is append-only and immutable. It also means current state must be materialised somewhere for fast reads, so you carry both the event log and one or more derived views and keep them consistent. This is a real design cost, and event sourcing is not the right pattern for every system. But where auditability, temporal queries and the ability to rebuild state matter, the events-as-truth model is genuinely transformative rather than merely fashionable.

A closely related technique is change data capture, where you turn the existing changes in a conventional database into an event stream without rewriting the application. That is how many organisations get the benefits of streaming without a full event-sourcing rebuild, and it is covered in the piece on change data capture.

3. Streams versus queues versus batch

People new to streaming often assume it is just a faster message queue, and that conflation causes real design mistakes. It helps to line up the three main ways of moving data between systems and see clearly what each one does.

Batch processing collects data over a window, an hour, a night, a month, and processes it all at once. It is efficient, simple to reason about, and perfectly appropriate for work that does not need to be current, such as monthly billing runs or overnight warehouse loads. Its defining limitation is latency: nothing downstream knows about a change until the next batch runs, so a business built on batch always operates on data that is at least one window stale.

Message queues move individual messages between systems in near real time and are excellent at distributing work. A classic queue, though, treats a message as something to be delivered once and then removed. When a consumer reads a message and acknowledges it, the message is typically gone. That is exactly what you want for task distribution, but it means a queue is not a system of record and generally does not let a second, later consumer replay the same messages. For the request-and-notify style of messaging, the companion article on publish and subscribe goes deeper.

Event streams sit between and beyond these. Like a queue they move events in near real time, but like a database they retain those events durably and in order, so many consumers can read them independently and new consumers can replay from the beginning. A stream is both a transport and a store. That dual nature is precisely why streaming platforms have become the backbone of real-time architectures: you get the low latency of messaging and the replayable durability of a log in one place.

A caution worth stating early: streaming is not automatically better than batch. Real-time processing carries real operational cost, more moving parts, harder debugging, and a standing infrastructure that must be kept healthy around the clock. If a business process genuinely runs on a daily or monthly rhythm, a well-built batch job is cheaper, simpler and easier to reason about. Reach for streaming when the value of acting on data quickly outweighs that added complexity, not because streaming sounds more modern.

4. How stream processing works

In a streaming architecture, data flows from producers on the left, through a central streaming platform in the middle, out to independent consumers and processors on the right. The platform holds the events in ordered, partitioned logs called topics, and each consumer tracks its own position in the log so it can read at its own speed without affecting anyone else. The diagram below shows the shape of a typical flow.

Stream processing is the layer that does real work on events as they flow, rather than after they land. A stream processor reads from one or more topics, transforms or enriches each event, and often writes results back to another topic or to a database. The operations look familiar to anyone who has written SQL: filtering events, mapping fields, joining two streams together, and aggregating over windows of time. The difference is that the input never ends. A batch query runs over a fixed dataset and finishes; a streaming query runs forever over an unbounded input and emits results continuously.

The subtle part of stream processing is time and windows. Because events arrive continuously, aggregations such as "count of orders in the last five minutes" must be computed over a moving window, and events can arrive late or out of order, especially from mobile or IoT sources. Mature stream processors distinguish between the time an event actually happened and the time it was processed, and they let you define windows on event time so a late arrival is still counted in the correct bucket. Getting this right is what separates a toy streaming demo from a production system that produces correct numbers under real-world messiness. It is also where most of the genuine engineering effort in a streaming project goes.

5. Streaming versus traditional messaging

Because streaming platforms and message brokers both move events between systems, teams frequently pick the wrong tool by treating them as interchangeable. The table below compares event streaming with traditional messaging and with batch across the dimensions that actually drive the decision: timing, data model, retention and the work each is best suited to.

Read the table as a guide to intent rather than a ranking. The line that matters most is retention: a streaming platform keeps events and lets many consumers replay them, while a traditional broker generally delivers a message once and forgets it. If you need several systems to read the same events independently, or you need to add a new consumer next year that replays two years of history, streaming is the natural fit. If you simply need to hand a unit of work from one service to another and never look at it again, a queue is lighter and entirely sufficient.

6. Real use cases

Event streaming is not a solution looking for a problem; it earns its place in a handful of recurring situations where the flow of events is genuinely the thing that matters. Four categories cover most of what I see in practice.

• Real-time analytics: dashboards, operational metrics and alerting that need to reflect what is happening now rather than what happened at the last batch. A logistics operation watching delivery events, a retailer tracking live sales, a utility monitoring consumption; each reads a stream and updates continuously. The value is that a decision-maker sees a developing situation while there is still time to act on it, not the morning after.
• Event sourcing: applications that use the event log as their system of record, deriving current state by replaying events. This gives a complete audit trail, temporal queries and the ability to rebuild any read model on demand. It suits domains where correctness and history are paramount, finance, ledgers, order lifecycles, and where the ability to reconstruct exactly how state was reached has real business or regulatory value.
• Change data capture pipelines: turning the commit stream of an existing operational database into events, so that changes flow to a warehouse, a search index or another service in near real time without nightly extracts. CDC is often the pragmatic on-ramp to streaming because it delivers real-time data movement without rewriting the source application at all.
• IoT telemetry: fleets of sensors and devices emitting readings continuously, where the volume and velocity make batch impractical and the value of the data decays quickly. Streaming ingests the telemetry, processes it near the point of arrival, aggregates it over time windows and feeds both live monitoring and longer-term storage. This is a natural fit because the source is already a stream by its nature.

What ties these together is that each treats the arrival of an event as the trigger for work, and each benefits from many consumers reading the same events for different purposes. When you find yourself writing multiple polling jobs against the same table, or copying the same data to several destinations on overlapping schedules, that is usually the signal that a stream would serve you better than a pile of batch jobs. In the Microsoft stack specifically, the way business events surface through APIs and webhooks is worth understanding alongside this; the piece on Business Central APIs and integrations covers that ground.

7. Common platforms

The best known event-streaming platform is Apache Kafka, an open-source system originally built at LinkedIn and now maintained under the Apache Software Foundation. Kafka popularised the model this whole article describes: a distributed, partitioned, replicated commit log where producers append events to topics and consumers read at their own offsets, with events retained for a configurable time so history can be replayed. Its durability, throughput and ecosystem made it the default choice for large-scale streaming, and much of the industry vocabulary, topics, partitions, offsets, consumer groups, comes from it. The dedicated companion piece, Kafka explained, goes into how it works in detail.

Around and beyond Kafka sit several other options. The major cloud providers offer managed streaming services so teams can consume the pattern without operating the infrastructure themselves. There are also stream-processing frameworks, distinct from the transport layer, that specialise in the compute side, running the continuous filtering, joining and windowed aggregation over events that section four described. And some traditional message brokers have added streaming-style retention features, blurring the old line between a queue and a log. The right choice depends less on brand and more on whether you need a durable replayable log, heavy stream-processing compute, or simple message transport, and on whether you want to run it yourself or buy it as a managed service.

My advice to clients is to separate the two questions cleanly. First decide whether you actually need streaming semantics, ordered, retained, replayable events consumed by many independent readers, or whether a plain queue or a batch job would do. Only once that is settled should you compare platforms. Picking a streaming product before you have confirmed you need streaming is one of the most common and most expensive ways these projects go wrong.

8. When to use event streaming

Event streaming is powerful, and like most powerful things it is easy to over-apply. The honest test I use is whether the flow of events, and the ability to react to them quickly and to replay them later, carries enough value to justify the standing complexity of a real-time platform. A few conditions point clearly toward streaming:

• Many consumers of the same data: when several systems all need the same changes for different purposes, a single stream that each reads independently beats a tangle of point-to-point feeds and overlapping polling jobs.
• Latency genuinely matters: when acting within seconds rather than hours changes the outcome, fraud interception, live monitoring, dynamic pricing, operational alerting, the real-time nature of streaming pays for itself.
• History and replay have value: when you need an audit trail, temporal queries, or the ability to build new views by reprocessing past events, the retained log is worth carrying.
• The source is naturally a stream: telemetry, clickstreams, transaction feeds and device data arrive as continuous events already, and forcing them into batch windows discards exactly the timeliness that makes them valuable.

And a few conditions point away from it. If a process runs on a daily or monthly rhythm and nobody needs the data sooner, batch is cheaper and simpler. If you only ever hand one unit of work from one service to one other and never replay it, a queue is lighter. If your team has no operational capacity to run and debug a distributed, always-on platform, adopting streaming for a marginal benefit will cost more in reliability than it returns in freshness. Streaming is a commitment, not a free upgrade, and the discipline to say no to it where it does not belong is as valuable as the ability to build it where it does.

9. References

For readers who want to go deeper into the foundations rather than a vendor's marketing, two bodies of material are worth knowing.

• The Apache Kafka documentation, published by the Apache Software Foundation, is the primary reference for the streaming-log model, covering topics, partitions, offsets, consumer groups, retention and the stream-processing library. It is the canonical source for how a durable, replayable event log actually behaves, and much of the vocabulary in this article traces back to it.
• The event-sourcing and CQRS literature, developed over many years by practitioners in the domain-driven design community, sets out the pattern of using an immutable log of events as the source of truth and deriving read models from it. This body of writing is where the reasoning behind events-as-truth, compensating events and materialised views is worked out carefully rather than asserted.

Rather than link to deep pages that move over time, I would point you to the official Apache Kafka documentation on its project site and to the established writing on event sourcing and CQRS from the design community; both are easy to find by name and are more durable references than any single article link.

Final thoughts

Event streaming is, at its core, a change of emphasis. It says the interesting thing about a business is not the current value of a row but the sequence of events that produced it, and that if you capture that sequence faithfully, ordered, durable and replayable, a great many integration and analytics problems become tractable. Systems decouple, new consumers join without disturbing old ones, history becomes an asset rather than something overwritten, and the organisation can react to what is happening while it is still happening.

None of that makes streaming the right answer everywhere. It is a real capability with a real operational cost, and it belongs where events flow continuously, where several systems need the same data, where latency changes outcomes, and where the retained log earns its keep. Applied there it is genuinely transformative. Applied to a process that runs happily on a nightly batch, it is complexity you will pay for and not use. The skill, as with most of enterprise integration, is knowing the difference, and for the wider map of how all these patterns fit together, the pillar on enterprise system integration is the place to keep reading.

Related reading: Enterprise system integration explained, Kafka explained, What is publish and subscribe, What is change data capture, Business Central APIs and integrations.