Integration Error Handling Best Practices

Building the happy path of an integration is the easy part. Two systems agree on a format, you map the fields, you send the message, it arrives, everyone celebrates. Then the network hiccups, the target system restarts mid-batch, a payload arrives with a field the other side rejects, and the integration that demoed beautifully starts losing records in ways nobody can explain. In more than two decades of connecting ERP, EAM, CAFM and finance systems, I have learned that error handling is not a feature you bolt on at the end. It is the difference between an integration that runs for years untouched and one that becomes a permanent source of firefighting. This guide walks through the patterns that make integrations resilient, and it assumes you already understand the fundamentals covered in the enterprise system integration pillar.

The message up front: resilient integration is not about preventing failure, which is impossible, it is about failing in a controlled, observable, recoverable way. Every pattern in this article exists to answer one question: when this call fails, what happens next, and does anyone lose data or trust in the process?

1. Why error handling makes or breaks integrations

An integration is a promise between two systems that a piece of information will move reliably from one to the other. The moment you make that promise across a network, you have inherited every way a network and a remote system can fail: timeouts, dropped connections, rate limits, authentication expiry, malformed responses, partial outages, and the target system simply being down for maintenance. None of these are exotic. On a busy integration they happen every day. The question is never whether they happen, it is whether your integration treats them as expected events with a defined response, or as surprises that corrupt data and stop the flow.

The integrations that fail in production are rarely the ones with clever logic. They are the ones with naive error handling: a single try block that catches everything and logs "error occurred", a retry loop with no limit that hammers a struggling system until it collapses entirely, a batch job that stops dead on the first bad record and leaves the other nine thousand unprocessed. These are not edge cases discovered years later. They are predictable consequences of treating error handling as an afterthought. The cost lands as silent data loss, duplicated transactions, cascading outages and, worst of all, a loss of confidence from the business users who now check every figure by hand because they no longer trust the integration.

The reframe I push with every team is this: design the failure paths with the same care you give the happy path. For every call your integration makes, you should be able to answer what happens on a timeout, on a rejection, on a duplicate, and on the target being unavailable for an hour. If you cannot answer those four questions, the integration is not finished, it is merely demonstrated. Getting this right is also a recurring theme in the common integration mistakes pillar, where poor error handling shows up as the root cause behind most of the disasters.

2. Transient versus permanent errors

The single most important distinction in error handling is whether a failure is transient or permanent, because it dictates the entire response. Confuse the two and you will either retry things that can never succeed, or give up on things that would have worked on the next attempt.

Transient errors are temporary and self-correcting. The connection timed out because the network was briefly congested. The target returned a 503 because it was restarting or under momentary load. A rate limit was hit and will reset in thirty seconds. A database deadlock rolled back a transaction that would commit cleanly if tried again. The defining property of a transient error is that the same request, sent again a moment later, has a good chance of succeeding. Transient errors are the domain of retries.

Permanent errors are deterministic. The payload is missing a mandatory field. The target rejected the record because a foreign key does not exist. Authentication credentials are invalid. The requested resource does not exist. A business rule on the far side forbids the operation. The defining property of a permanent error is that retrying the identical request will fail identically every time, forever. Retrying a permanent error is not just useless, it is harmful: it wastes resources, delays the batch, and floods your logs while masking the real problem, which is that a human or an upstream fix is required.

In HTTP terms, and this is a rough guide rather than a rule, most 5xx responses and connection-level failures are candidates for retry, while most 4xx responses are permanent and should not be retried. A 429 (too many requests) is the notable transient exception among the 4xx codes, and a 409 (conflict) often signals a duplicate you should handle rather than retry. The practical discipline is to classify every possible failure your integration can encounter into transient or permanent up front, and route each class to the right pattern. Transient goes to retry-and-backoff. Permanent goes to the dead-letter queue and an alert, because a human needs to look at it.

3. Retries and exponential backoff

Retrying is the correct response to a transient error, but naive retrying is one of the most dangerous things you can do to a distributed system. The failure mode is well known and has a name: the retry storm. A downstream service slows down or briefly falls over. Every caller immediately retries. The flood of retries lands on the already-struggling service at the exact moment it can least handle load, and instead of recovering it collapses completely. The retries meant to improve reliability have amplified a small blip into a full outage.

The fix is exponential backoff: instead of retrying immediately and at a fixed interval, you wait progressively longer between attempts. A typical schedule doubles the delay each time, for example one second, then two, then four, then eight, giving the struggling service room to breathe and recover before the next attempt. You also cap the number of retries, because a transient error that has not cleared after several attempts is behaving like a permanent one and should be escalated rather than retried indefinitely.

Backoff alone still has a subtle flaw. If a hundred callers all failed at the same instant and all use the same doubling schedule, they will all retry at the same instants too, arriving in synchronised waves. The refinement is jitter: add a small random variation to each delay so the retries spread out across time rather than clustering. Exponential backoff with jitter is the standard, well-tested combination, and it is what I recommend as the default for any retry logic that talks to a shared service.

The diagram below shows how retries, backoff, a dead-letter queue and a circuit breaker fit together into one coherent error-handling flow. A failed call is retried with growing delays; if it keeps failing it is moved to a dead-letter queue and an alert is raised; and a circuit breaker sits in front of the caller so that a persistently failing target stops new calls from even being attempted until it recovers.

4. Idempotency: making retries safe

Retrying is only safe if repeating an operation does no harm, and that property has a name: idempotency. An operation is idempotent if performing it twice produces the same result as performing it once. Reading a record is naturally idempotent. Setting a value to a fixed amount is idempotent. The dangerous case is anything that increments, appends or creates: "post a payment of 500", "add a line item", "create a work order". Retry one of those after a timeout and you may well end up with two payments, two line items, two work orders, because the first call actually succeeded and only the response was lost.

This is the trap that makes retries hazardous. A timeout does not tell you whether the operation failed or whether it succeeded and the acknowledgement got lost on the way back. If you retry blindly, and the original succeeded, you have now duplicated it. On a financial or transactional integration, silent duplication is far worse than an obvious failure, because nobody notices until the numbers are wrong.

The standard defence is an idempotency key: the caller attaches a unique identifier to each logical operation, and the receiver records which keys it has already processed. When a retry arrives carrying a key the receiver has already seen, it recognises the duplicate and returns the original result instead of performing the operation again. The operation becomes safe to retry any number of times because only the first occurrence has an effect. Where you cannot control the receiver, you achieve the same protection by designing operations to be naturally idempotent, for example using "set status to approved" rather than "advance status by one step", so that repetition is harmless.

The honest limitation: idempotency is not free and it is not automatic. Someone has to generate stable keys, the receiver has to store and check them, and that store needs a retention policy so it does not grow forever. Teams routinely add retries for resilience and forget idempotency, and then discover months later that every network blip has been quietly creating duplicate records. If you introduce retries on any create-or-modify operation, idempotency is not optional, it is the other half of the same decision.

5. Dead-letter queues

Some failures are not transient and will never succeed on retry. A message references a customer that does not exist, or carries a malformed date, or violates a business rule on the receiving side. You cannot retry your way past these, and you must not let them block the flow. If a batch or a queue stops dead on the first unprocessable message, one bad record holds up thousands of good ones, and a minor data-quality issue becomes a full outage.

The pattern that solves this is the dead-letter queue, often abbreviated DLQ. When a message has exhausted its retries or is recognised as permanently unprocessable, it is moved aside into a separate queue rather than discarded and rather than left to block the pipeline. The main flow continues processing everything else, and the parked messages sit in the dead-letter queue with enough context, the original payload, the error, the timestamp, the number of attempts, for a human to investigate, fix the underlying cause, and either correct and reprocess them or reject them deliberately.

A dead-letter queue turns silent data loss into visible, recoverable exceptions. Without it, the two common outcomes are both bad: either the bad message blocks everything behind it, or the integration swallows it and the record simply vanishes with no trace. With a DLQ, nothing is lost, nothing is blocked, and there is a clear worklist of exceptions to work through. This is a core capability of any serious message broker, and it pairs naturally with queue-based integration; the mechanics of queues themselves are covered in the message queue pillar. The discipline that matters most in practice is that a dead-letter queue is not a place messages go to be forgotten. An unmonitored DLQ that quietly fills up is just a slower form of data loss, which is why it must always be paired with alerting, covered further below.

6. Circuit breakers

Retries protect against brief blips, but they are the wrong tool when a target system is genuinely down. If a service has been failing for two minutes, continuing to fire requests and retries at it achieves nothing except wasting resources on your side and piling load onto a system that is already struggling. What you want is to stop calling it, give it time to recover, and fail fast in the meantime so that your own system is not tied up waiting on doomed calls.

That is exactly what the circuit breaker pattern provides, borrowing its name and its logic from an electrical circuit breaker that trips to protect a circuit from damage. The breaker sits between the caller and the target and watches the outcomes of calls. It has three states. In the closed state, calls flow through normally and the breaker simply counts failures. When failures cross a threshold, say a certain percentage within a window, the breaker trips open. In the open state, calls are rejected immediately without even attempting to reach the target, failing fast and sparing both systems. After a cooldown period the breaker moves to half-open, where it lets a small number of trial calls through. If those succeed, the target has recovered and the breaker closes again; if they fail, it trips back open and waits longer.

The value of a circuit breaker is twofold. It protects the failing downstream service from being hammered while it is trying to recover, and it protects the caller from tying up threads, connections and memory on calls that are going to fail anyway. Without a breaker, a single slow or dead dependency can exhaust the caller's resources and drag down services that had nothing to do with the original fault, which is how one component's outage cascades into a system-wide one. The circuit breaker is the pattern that contains that blast radius, and in the flow diagram above it is the dashed boundary guarding the caller.

7. Compensating transactions and sagas

Everything so far assumes a single operation between two systems. Real business processes often span several systems, and there you hit a harder problem: you cannot wrap a call to a finance system, an EAM system and a warehouse system in one database transaction that either all commits or all rolls back. There is no distributed transaction to lean on across independent systems and APIs. So what happens when step one and step two succeed but step three fails, leaving the process half-completed and the systems inconsistent?

The answer is the compensating transaction, coordinated by what is known as the Saga pattern. Instead of one atomic transaction, you model the process as a sequence of local steps, each with a defined compensating action that semantically undoes it. If a later step fails, you do not roll back a shared transaction, because there is none; instead you run the compensating actions for the steps that already succeeded, in reverse order, to bring the whole process back to a consistent state. Reserve stock, then charge the customer, then create the shipment: if the shipment step fails, you compensate by refunding the charge and releasing the reserved stock. Each undo is itself a normal business operation, not a technical rollback.

Compensating transactions are more work to design than a simple retry, because you have to define the reverse of every step and reason carefully about partial failure at each stage. But for multi-system business processes they are the only honest way to maintain consistency without pretending you have a distributed transaction you do not have. The key discipline is to make each step and each compensation idempotent, so that retries and re-runs during recovery do not themselves create new inconsistencies. This is where idempotency and sagas reinforce each other.

8. Alerting and manual intervention

Every pattern above eventually reaches a point where automation has done all it safely can and a human must step in. Retries have been exhausted, a message has landed in the dead-letter queue, a circuit breaker has been open for an hour, or a saga has compensated and needs someone to understand why. The pattern that makes all of this real is alerting, and it is the piece teams most often skimp on, because it produces nothing on the happy path and only proves its worth on a bad day.

Good alerting is specific, actionable and proportionate. It tells the right person what failed, in which integration, how many records are affected, and where to look, and it does so without burying that signal under a flood of noise. The two failure modes are equally damaging. Too few alerts and problems accumulate silently until someone notices the numbers are wrong days later. Too many alerts and the team learns to ignore them, so the one that actually matters scrolls past unread amid the routine noise. The craft is in alerting on conditions that genuinely require human action, a dead-letter queue that is growing, a circuit breaker that has not closed, a retry rate that has spiked, and staying quiet about the transient blips the system is designed to absorb on its own.

Alerting is inseparable from observability more broadly. You cannot alert on what you do not measure, and you cannot investigate an alert without logs and metrics that let you trace a failed message from source to destination. Correlation identifiers that follow a message across every hop, structured logs, and dashboards that show retry rates and queue depths are what turn an alert from "something is wrong" into "here is exactly which records failed and why". That full observability discipline is the subject of the integration monitoring and logging pillar, and it is the foundation that makes every error-handling pattern in this article operable rather than theoretical.

The manual-intervention workflow deserves as much design attention as the automated paths. Someone needs a clear way to see what is in the dead-letter queue, understand each failure, correct the data or the upstream cause, and reprocess or reject the parked messages. An integration that produces exceptions but gives operators no tools to resolve them has simply relocated the firefighting rather than reduced it.

9. The patterns summarized

Each of these patterns solves a specific failure mode, and a robust integration usually combines several of them. The table below sets out what each pattern does and when to reach for it, as a quick reference you can hold against your own integration design.

These patterns are not alternatives to choose between; they are layers that stack. A well-built integration retries transient failures with exponential backoff, keeps those retries safe with idempotency, protects itself and its dependencies with a circuit breaker, routes the genuinely unprocessable to a dead-letter queue with an alert, and, where a process spans systems, uses compensating transactions to keep them consistent. The specifics vary by platform. If you are integrating Microsoft Dynamics 365 Business Central, for example, the same principles apply on top of its API model, as covered in the Business Central APIs and integrations pillar.

The one to internalise: classify every failure as transient or permanent, then send transient failures to retry-with-backoff-and-idempotency and permanent failures to a dead-letter-queue-with-alert. Put a circuit breaker in front of anything that can go down, and use compensating transactions when a process crosses systems. Get that routing right and the rest is detail. For the wider context these patterns sit inside, keep the enterprise system integration pillar close at hand.

10. References

The patterns in this article are long-established in the distributed-systems and enterprise-integration literature rather than anything I invented, and it is worth knowing the canonical names so you can research each one further.

• The Circuit Breaker pattern was popularised by Michael Nygard in his book on building production-ready software, and is now a standard resilience pattern documented across major cloud and integration platforms.
• The Saga pattern for managing consistency across distributed transactions traces back to academic work on long-lived transactions in the late 1980s and has become the standard approach for coordinating compensating transactions in microservice and multi-system architectures.
• Retry with exponential backoff and jitter is documented as a standard practice by every major cloud provider and messaging platform as the safe way to handle transient faults.
• Enterprise Integration Patterns, the pattern catalogue by Gregor Hohpe and Bobby Woolf, is the foundational reference for messaging concepts including dead-letter channels and message-based error handling.

None of these require a specific product. They are design patterns you can apply whether you build on an enterprise service bus, a cloud integration platform, a message broker, or hand-written code, and the value comes from applying them deliberately rather than from any tool that claims to provide them out of the box.

Final thoughts

Resilient integration is a mindset before it is a set of patterns. The mindset is that failure is normal, expected and continuous, and that the job of an integration is not to avoid failure but to absorb it invisibly, recover from it automatically where it can, and escalate it clearly where it cannot. The patterns in this guide are the concrete expression of that mindset. Retries and backoff absorb the transient. Idempotency keeps that absorption safe. Dead-letter queues and alerting make the permanent visible and recoverable. Circuit breakers contain the blast radius of a downstream outage. Compensating transactions hold multi-system processes consistent when there is no transaction to lean on.

The integrations I have seen run untouched for years all share the same trait: their builders treated the failure paths as first-class design, not as exceptions to be swept into a catch-all handler. The ones that became a permanent maintenance burden all share the opposite trait. If you take one thing from this article, let it be the habit of asking, for every call your integration makes, what happens when this fails, and building the answer in from the start. That single discipline, applied everywhere, is what turns a brittle demo into an integration the business can actually trust.

Related reading: Enterprise system integration explained, Integration monitoring and logging, Common integration mistakes, What is a message queue, Business Central APIs and integrations.