Batch vs Real-Time Integration

If you have read the enterprise system integration pillar, you already know that integration is the plumbing that lets separate systems share data and behave as one. This article zooms in on a single, recurring decision inside that plumbing: should data move in scheduled bulk loads, or should each change flow through the instant it happens? Every integration you will ever design falls somewhere on that spectrum, and the two ends have very different costs, failure modes and operational demands. Getting this decision right is the difference between an integration that quietly earns its keep for a decade and one that either misses the business need or burns money keeping data fresh that nobody looks at until Monday.

The message up front: batch and real-time are not better and worse versions of the same thing. They are different tools for different latency requirements. The right question is never "how fresh can we make the data" but "how fresh does the business actually need it, and what is the cheapest, most reliable way to deliver exactly that." Over-engineering for real-time is as much a design failure as under-delivering with batch.

1. The core difference

Strip away the vendor language and the difference comes down to one thing: when data moves.

Batch integration collects changes over a period of time and moves them together on a schedule. A payroll export runs at 2am and pushes the day's timesheet records to finance. A nightly job reads every new and changed customer record and loads it into the data warehouse. A weekly file drops all updated supplier prices into the purchasing system. The defining trait is that data is accumulated and then moved in bulk at defined intervals. Between runs, the target system is deliberately stale, and that staleness is accepted because the business does not need the data any sooner.

Real-time integration moves each change individually, the moment it occurs, as an event. A customer updates their address in the CRM and, within a second or two, the billing system, the delivery system and the loyalty platform all know. A sensor reports a temperature breach and an alert fires immediately. A payment clears and the order is released for fulfilment without anyone waiting for a scheduled job. The defining trait is that a single change triggers an immediate flow, with no waiting for a batch window.

Everything else that people argue about, file transfers versus APIs, message queues versus database extracts, hourly versus nightly, is really a variation on this one axis of latency. Batch trades freshness for simplicity and throughput. Real-time trades simplicity and throughput for freshness. That single trade-off is what this entire article is about, and understanding it clearly makes most integration design decisions obvious.

2. How batch integration works

Batch integration follows a rhythm that has barely changed in principle since mainframes, even though the tooling has modernised completely. A scheduler wakes a job at a defined time. The job identifies the records that are new or changed since the last run, usually by timestamp, a change flag, or a full comparison. It extracts those records, optionally transforms them into the shape the target expects, and loads them into the destination. Then it records how far it got so the next run knows where to resume, and it sleeps until the next scheduled window.

Most batch integration is some form of the extract, transform, load (ETL) pattern: pull data out of the source, reshape it, and push it into the target. The classic nightly data warehouse load is pure batch ETL. The diagram below shows the shape of a typical scheduled batch flow, where changes pile up in the source system all day and then move together in a single window.

The strengths of batch are exactly the strengths of doing work in bulk on your own schedule. You process large volumes efficiently, because moving a million rows in one connection is far cheaper per row than moving them one at a time. You run heavy work in quiet hours, so the transformation load does not compete with daytime users. You get a natural, well-defined recovery point: if the 2am run fails, you fix it and rerun the whole window, and the logic for "what changed since last success" is simple. And the whole thing is easier to reason about, test and audit, because it happens at known times with a clear start and end.

The cost of batch is equally clear: between runs, the target is stale. For a data warehouse that reports on yesterday, that is completely acceptable. For a customer checking whether their payment cleared, a data set that is hours old is useless. Batch is the right tool precisely when that staleness does not hurt, and the wrong tool the moment it does.

3. How real-time integration works

Real-time integration inverts the model. Instead of a scheduler deciding when data moves, the data itself decides, because a change in the source is what triggers the flow. The moment a record is created or updated, the source emits an event, and that event is carried to every interested system through an API call, a webhook, or a message broker. There is no waiting for a window, because there is no window. The unit of work is a single change, and the goal is to deliver it end to end in seconds or less.

Two broad mechanisms deliver real-time. The first is synchronous request and response, typically an API call: system A needs something from system B, calls its API, and waits for the answer. This is how a checkout page confirms a payment or a stock level in the moment. The Business Central APIs and integrations pattern is a good example of this synchronous, on-demand style, where one system queries another live rather than waiting for a nightly file. The second mechanism is asynchronous events, where a change is published to a broker and any number of downstream systems consume it independently. This is the domain of event streaming, where a continuous flow of events is the backbone of the architecture and new consumers can subscribe without the source ever knowing they exist.

The strength of real-time is freshness and responsiveness. Data is current everywhere, almost immediately. Systems react to events as they happen rather than discovering them hours later. Decoupled event flows let you add new consumers without touching the producer. This is what makes modern, responsive digital experiences possible, and there is a whole set of real-time integration use cases where nothing else will do.

The cost of real-time is complexity and continuous operational load. You are no longer running one well-understood job at 2am; you are running a stream of small operations around the clock, every one of which can fail independently. You need to handle retries, out-of-order delivery, duplicate events, partial failures where two of three targets updated and the third did not, and back-pressure when a downstream system cannot keep up. The infrastructure, a broker or event platform, must be highly available because it is now in the critical path of your business at all hours. None of this is exotic, but all of it is real work, and it is work you take on the moment you commit to real-time.

The honest caution: real-time integration multiplies your failure surface. A nightly batch job that fails is a contained problem you notice in the morning and rerun. A real-time flow that fails can silently drop or duplicate individual events all day, and the damage is spread across thousands of small transactions that are far harder to detect and reconcile. Do not choose real-time for the resume line. Choose it because the business genuinely cannot tolerate the latency of batch, and then budget properly for the monitoring, retry logic and reconciliation it demands.

4. Head to head

The clearest way to hold the two approaches in view is side by side. The table below compares them across the dimensions that actually drive the decision. Read it not as a scorecard where one column wins, but as a map of which approach fits which requirement.

The pattern in the table is consistent. Batch wins on simplicity, throughput efficiency and cost. Real-time wins on freshness and responsiveness. Neither column is the winner in the abstract, because the only dimension that matters for a given integration is the one the business requirement points at. If the requirement is "the warehouse must reflect yesterday accurately," every batch advantage applies and every real-time advantage is wasted. If the requirement is "block the fraudulent transaction before it completes," the reverse is true.

5. When batch is the right choice

Batch is the correct, and often superior, choice far more often than the industry's fascination with real-time would suggest. It is the right tool whenever the business can tolerate some staleness and you value simplicity, throughput and cost control. The clearest signals that batch is the answer:

• Reporting and analytics. A data warehouse or business intelligence layer that reports on completed periods does not need up-to-the-second data. Loading it nightly is cheaper, simpler and entirely sufficient. Forcing real-time freshness onto a reporting workload is pure over-engineering.
• Large-volume, low-urgency movement. Payroll, billing runs, statement generation, end-of-day reconciliations, and bulk data migrations all involve moving a lot of records where a few hours of latency is irrelevant. Bulk processing is dramatically more efficient than event-by-event handling for these.
• Periodic external files. Many partners, banks and government systems still exchange data as scheduled files. If the source only produces a daily file, real-time on your side buys nothing, because the data does not exist any sooner.
• Workloads that benefit from a clean recovery point. When correctness and auditability matter more than immediacy, the well-defined start and end of a batch window, and the ability to rerun it cleanly, is a genuine operational advantage.
• Cost-sensitive integrations. Batch runs on modest infrastructure at quiet hours. When the business value does not justify always-on event infrastructure, batch delivers the outcome for a fraction of the operating cost.

The discipline here is to resist the pull toward real-time for its own sake. A great deal of integration is reporting, reconciliation and bulk movement where batch is not a compromise but the right answer. Choosing batch confidently for those cases frees budget and engineering attention for the smaller set of flows that genuinely need real-time.

6. When real-time is worth the cost

Real-time earns its extra complexity and cost when the value of the data decays quickly, when a delay changes the outcome. The signals that real-time is the right choice:

• The decision is time-critical. Fraud detection, payment authorisation and safety alerting only have value if they happen before the transaction or the incident completes. A batch answer that arrives hours later is no answer at all.
• A human is waiting. When a customer or an operator is looking at a screen expecting the system to reflect what just happened, order status, account balance, inventory availability, latency is felt directly and staleness reads as a broken system.
• Multiple systems must stay in step continuously. Inventory shared across an e-commerce site, stores and a warehouse has to agree in near real-time, or you oversell stock you do not have. The cost of inconsistency is higher than the cost of streaming.
• Events drive downstream automation. When one change is supposed to trigger a chain of automated actions, releasing an order, notifying a courier, updating a loyalty balance, doing that on a nightly cycle defeats the purpose.
• You are building an event-driven architecture deliberately. When decoupling and extensibility are architectural goals, a real-time event backbone lets new consumers subscribe without disturbing producers, and that flexibility is worth the operational investment.

The common thread is that latency has a business cost that exceeds the engineering cost of eliminating it. When you can point at a concrete consequence of being minutes or hours behind, and that consequence is expensive, real-time is justified. When you cannot, it usually is not.

7. Micro-batch and near-real-time

The batch versus real-time framing is useful, but the real world is a spectrum, and a great deal of practical integration lives in the middle. Micro-batch runs the batch model on a very short cycle, every minute or every few minutes, rather than nightly. It keeps the operational simplicity of batch, a defined window, a clean rerun, bulk efficiency, while dramatically reducing staleness. For many workloads that feel like they need real-time, a five-minute micro-batch delivers all the freshness the business actually requires at a fraction of the complexity of true event streaming.

Near-real-time is the term for flows that are event-driven but do not guarantee sub-second delivery, where a few seconds of latency is acceptable. Most systems people call "real-time" are in fact near-real-time, and that is usually fine, because the business rarely needs true millisecond immediacy. Change data capture, where the source database's change log is read continuously and streamed to targets, is a common near-real-time technique that gives event-level freshness without asking the source application to emit events itself.

The practitioner's insight: the most common design mistake I see is treating this as a binary between nightly batch and full event streaming, then reaching for the expensive end because nightly feels too slow. Before committing to a streaming platform, ask whether a micro-batch every one to five minutes meets the need. Very often it does, and it does so with far less infrastructure, fewer failure modes and lower running cost. Reach for true real-time only when even a few minutes of latency carries a real business cost. Match the mechanism to the requirement, not to the ambition. If you want the wider context for where this decision sits, the enterprise system integration pillar lays out the full landscape.

8. How to choose

The decision is not about which approach is more modern or more impressive. It is a straightforward matching exercise, and asking these questions in order will settle it for almost any integration:

• How stale can the data be before someone is harmed or a decision goes wrong? This is the single most important question. If the honest answer is "a day is fine," you have a batch problem. If it is "seconds," you have a real-time problem. Most requirements sit somewhere in between, and that answer alone eliminates half the design space.
• What is the concrete cost of latency? Name the actual consequence of being an hour behind. If you cannot name a real, expensive consequence, you do not need real-time, regardless of how appealing it sounds.
• How much data moves, and how often does it change? High volume with infrequent, tolerable-latency change favours batch. A steady stream of individually meaningful changes favours real-time.
• Can the source even produce data any sooner? If the upstream system only generates a daily file, real-time on your side is wasted effort. Design to the constraint that actually exists.
• Do you have the operational maturity to run real-time reliably? Always-on event infrastructure needs monitoring, retry handling and reconciliation. If that capability is not in place, a solid micro-batch will serve the business better than a fragile real-time flow.
• Would micro-batch close the gap? Before committing to streaming, check whether a short-cycle batch delivers the freshness the business needs. It frequently does, at much lower cost and risk.

Work through those questions honestly and the answer usually chooses itself. The failures I am called in to fix are rarely a case of picking a mechanism that could not physically do the job. They are almost always a mismatch: real-time infrastructure built and maintained at real cost for a reporting workload that a nightly job would have served perfectly, or a nightly batch quietly starving a customer-facing feature that needed freshness the design never gave it. The mechanism was capable; it was pointed at the wrong requirement. Get the requirement clear first, and the batch-versus-real-time choice stops being a debate and becomes a decision.

9. References

The concepts in this article, batch and real-time processing, ETL, event-driven architecture, change data capture and the latency trade-offs between them, are well-established in the data-integration field. Readers who want to go deeper into the vendor-neutral foundations will find these primary sources useful:

• Amazon Web Services, "Batch Processing vs Stream Processing" and "What is ETL?" documentation, which frame the scheduled-bulk versus continuous-flow distinction in cloud terms.
• Microsoft Azure Architecture Center, "Batch processing" and "Real time processing" guidance, which set out when each style fits and the operational demands of each.
• Martin Kleppmann, Designing Data-Intensive Applications (O'Reilly), for the underlying treatment of batch systems, stream processing and change data capture.
• Confluent and Apache Kafka documentation, for the event-streaming and near-real-time patterns referenced in the real-time sections.
• Microsoft Dynamics 365 Business Central developer documentation, for the API-based, on-demand integration pattern used as a synchronous real-time example.

The judgements about where each approach pays off, and the recurring mismatch failures, are drawn from my own integration work across ERP, EAM and CAFM platforms rather than from any single published source.

Final thoughts

Batch and real-time integration are not rungs on a ladder where real-time is the top. They are two tools shaped for two different latency requirements, and the entire skill is matching the tool to the need. Batch trades freshness for simplicity, throughput and low cost, and it is the right answer for reporting, bulk movement and anything the business can tolerate being a little stale. Real-time trades simplicity and cost for immediacy, and it is worth every bit of that extra effort when a delay carries a real, expensive consequence. In between sits a large and underused middle ground of micro-batch and near-real-time that quietly solves most of the cases people assume demand full event streaming.

The best integration designers are not the ones who reach for real-time by reflex. They are the ones who ask how fresh the data actually needs to be, name the real cost of latency, and then deliver exactly that with the simplest, most reliable mechanism that meets it. Do that consistently and your integrations are cheaper to run, easier to trust, and quietly correct in a way nobody has to think about. Chase real-time for its own sake and you pay, in money and in failure modes, for freshness the business was never going to use.

Related reading: Enterprise system integration explained, What is ETL?, What is event streaming?, Real-time integration use cases, Business Central APIs and integrations.