Integration Monitoring and Logging

Integrations are the quiet plumbing of the enterprise. When they work, nobody thinks about them. When they stop, everything downstream stops with them, and the first person to notice is usually a user or a customer rather than the team that owns the pipe. Over two decades of building and rescuing enterprise integrations, I have learned that the difference between a mature integration estate and a fragile one is rarely the elegance of the code. It is whether the people running it can see what is happening inside the flows in real time. Monitoring and logging are not a nice-to-have you bolt on after go-live. They are part of the design, and if you skip them you are shipping a system that fails silently. This guide sits underneath the broader enterprise system integration pillar, and it focuses on one thing: how to make integrations observable so that problems announce themselves before your users do.

The message up front: observability is not about collecting more data, it is about being able to answer three questions fast. Is the integration healthy right now? If a specific transaction failed, exactly where and why did it fail? And is the trend getting better or worse? If your monitoring cannot answer those three questions in minutes, it is decoration, not instrumentation.

1. Why you cannot manage what you cannot see

The most dangerous integration is the one that fails without telling anyone. A synchronous API call that returns an error is, in a strange way, a well-behaved failure: someone gets an immediate signal that something went wrong. The failures that hurt are the quiet ones. A scheduled job that stops running because a credential expired. A message that lands in an error queue and sits there. A field that starts arriving null and gets written as a blank instead of raising an exception. These do not crash anything. They just slowly poison the data on both sides of the integration, and by the time the discrepancy surfaces in a report or a reconciliation, you are doing forensic archaeology across systems that have all moved on.

I have walked into more than one engagement where the brief was "the finance figures do not match" and the real problem, three layers down, was an integration that had been half-working for months. Nobody had built a way to see it. There was no dashboard, no alert, no log retention beyond a rolling few days, so there was no evidence trail. Rebuilding the timeline of what synced and what did not took longer than fixing the actual defect. That is the tax you pay for invisibility, and it compounds. Every day an unobserved integration runs is another day of potential silent corruption you will have to unwind later.

Observability flips the default. Instead of assuming an integration is fine until someone proves otherwise, you assume it must continuously prove itself healthy. The system emits a steady heartbeat of evidence: counts, timings, error rates, queue depths. When that evidence deviates from normal, you know before the business does. This is not exotic engineering. It is the same discipline a good maintenance team applies to physical assets, watching condition rather than waiting for breakdown, applied to the flows of data between systems. The mindset matters more than any particular tool. Decide that a silent integration is unacceptable, and the rest is implementation.

2. Logs, metrics and traces

Modern observability rests on three complementary kinds of signal, and understanding what each is good for stops you from trying to make one do the job of another. They are usually called the three pillars of observability, and I will name them properly in the References section, but the practical distinction is what matters here.

Logs are timestamped records of discrete events. "Received order 48213 from the web store at 09:14:02." "Posted invoice to the ERP, response 201." "Validation failed: customer tax ID missing." Logs are the narrative. When you need to know exactly what happened to one specific transaction, logs are where you look. Their strength is detail; their weakness is volume. Log everything at full verbosity and you generate a haystack so large that finding the needle becomes its own problem, and your storage bill quietly balloons.

Metrics are numbers aggregated over time. Messages processed per minute, average latency, error rate as a percentage, queue depth right now. Metrics are cheap to store because they are just numbers on a timeline, and they are what you watch continuously and alert on. Their strength is that they compress the state of the whole system into a handful of trends you can eyeball in seconds. Their weakness is that they tell you something is wrong without telling you which transaction or why. A spike in the error-rate metric says "go look"; the logs tell you what to look at.

Traces follow a single request as it travels across multiple systems and components. A trace stitches together the individual spans, one per hop, into a single end-to-end picture: the web store took 40 milliseconds, the integration layer took 120, the ERP took 800, the response came back. Traces are how you answer "where is the time going" and "at which hop did this transaction die" in a distributed flow. They are the pillar that most integration teams skip, and it is exactly the pillar that saves you when a problem spans three systems and each one insists the fault is somewhere else.

The three work together. Metrics tell you something is wrong and roughly when. Traces tell you where in the flow it went wrong. Logs tell you exactly what happened at that point. Build all three, wire them so you can jump from a metric spike to the relevant traces to the underlying log lines, and you have turned a black box into a system you can reason about. This layered picture is part of good integration architecture design from the start, not something to retrofit under pressure.

3. Correlation IDs and distributed tracing

Here is the single most valuable piece of instrumentation you can add to an integration, and it costs almost nothing: a correlation ID. When a transaction first enters your estate, generate a unique identifier for it, attach that identifier to every log line, every message, every downstream call it triggers, and propagate it through every system it touches. Now, when something goes wrong, you can filter every log in every system by one value and reconstruct the entire journey of that one transaction in seconds.

Without a correlation ID, tracing a failed order across a web store, a middleware layer, a message broker and an ERP is a manual cross-referencing exercise across four sets of logs with four different timestamp formats and no shared key. With one, it is a single search. The diagram below shows the shape of it: one correlation ID threading through every system while logs, metrics and traces flow out to a central place that feeds both the dashboard your team watches and the alerts that page them.

Distributed tracing is the formal version of this idea. Standards for context propagation define exactly how a trace identifier and its per-hop span identifiers travel in message headers so that tracing tools can assemble the end-to-end picture automatically. You do not have to adopt a heavyweight tracing platform on day one to get most of the value. Even a home-grown correlation ID, generated at the edge and logged everywhere, transforms your ability to investigate. Start there, and graduate to full distributed tracing when the estate is large enough to justify it. The rule I hold teams to is simple: no message should ever move through the estate without carrying an identifier that ties it back to the transaction that spawned it.

4. The key metrics to monitor

You cannot alert on everything, and you should not try. A small set of metrics captures the health of almost any integration, and disciplined attention to these beats a sprawling dashboard nobody reads. The table below is the core set I instrument on virtually every integration, along with why each one matters and what a healthy signal looks like. The specific thresholds are always tuned to the particular flow, but the shape of "healthy" is remarkably consistent.

Notice how several of these metrics are leading indicators rather than lagging ones. Queue depth and retries both start climbing before the error rate spikes, which gives you a chance to intervene before transactions actually fail. That is the whole point of instrumentation: to buy time. A team watching queue depth catches the slowdown while it is still just a slowdown; a team watching only the error rate catches it after customers are already affected.

5. Dead-letter queues and error visibility

A dead-letter queue is where messages go when they cannot be processed after all reasonable retries have been exhausted. It is one of the most important safety mechanisms in asynchronous integration, and it is also one of the most neglected. The pattern is sound: rather than blocking the whole pipeline on one poison message, or worse, discarding it silently, you move the failed message to a separate queue where it is preserved for inspection and reprocessing. The failure is contained without being lost.

The neglect comes in what happens next, which is often nothing. I have opened dead-letter queues on live systems and found thousands of messages going back months, each one a transaction that never completed, none of them ever looked at. The mechanism was working perfectly. The organisation around it was not. A dead-letter queue is only a safety net if someone is watching it. An unwatched one is just a quiet graveyard of failed business transactions, and every message in it represents data that is now inconsistent between the systems the integration was supposed to keep in sync.

The honest pitfall: teams treat a non-empty dead-letter queue as a normal operating state rather than an alert. It is not. A message in the dead-letter queue is a transaction that failed and will stay failed until a human acts. Depth greater than zero should page someone, and the messages should carry enough context, including the correlation ID and the original error, to diagnose them without re-running the whole flow. If you have to reproduce the failure to understand it, your error logging was insufficient.

Good error visibility means three things. Every failure is captured with enough context to diagnose it, ideally the full payload, the error, and the correlation ID. Failures are surfaced actively rather than waiting to be discovered. And there is a defined path to reprocess a corrected message rather than re-keying it by hand. How you handle the failure itself, retries, backoff, idempotency, is a subject in its own right; see the companion guide on integration error handling best practices for the patterns that sit alongside this visibility.

6. Alerting that works (avoiding alarm fatigue)

The failure mode of alerting is not too few alerts, it is too many. A team that gets paged forty times a night for conditions that resolve themselves learns, entirely rationally, to ignore the pager. Then the one alert that mattered arrives at 3am and gets swiped away with the rest. Alarm fatigue is how well-instrumented systems still suffer surprise outages: the signal was there, but it was buried in noise the team had trained itself to disregard.

The discipline that fixes this is to alert on symptoms that require human action, not on every deviation. The test I apply to every proposed alert is blunt: if this fires at 3am, is there something a person must do right now? If the honest answer is "no, it will sort itself out" or "we will look at it in the morning," it is not an alert, it is a dashboard metric or a daily report. Only conditions that demand immediate human intervention earn the right to page someone.

A few practices keep alerting sharp. Alert on sustained conditions rather than instantaneous spikes, so a single blip does not wake anyone but a five-minute sustained error rate does. Set thresholds against normal behaviour rather than arbitrary round numbers, because "normal" for a nightly batch looks nothing like "normal" for a real-time feed. Route alerts by severity, so a dead-letter message and a minor latency bump do not arrive through the same channel with the same urgency. And review every alert that fires: if it was not actionable, tune or delete it. An alert that has cried wolf three times has already lost its power, and keeping it does more harm than removing it.

7. Dashboards and service levels

Dashboards and alerts answer different questions and should not be confused. Alerts answer "does someone need to act right now." Dashboards answer "what is the state of the estate" for someone who is already looking. A good integration dashboard lets an operator glance and know, within a few seconds, whether every flow is healthy: throughput tracking expectations, error rates at baseline, queues shallow, no dead letters. It is the situational-awareness layer, and it is what you put on a wall or check at the start of a shift.

The trap with dashboards is the opposite of the trap with alerts. Where alerts fail by being too noisy, dashboards fail by being too busy. A dashboard with sixty panels is not a dashboard, it is a data dump, and the important signal hides among the trivia exactly as an over-alerted pager buries the important page. Design the top-level view to show only what tells you whether the system is healthy, and let people drill into detail from there. The first screen should be answerable in a glance; depth lives one click down.

Behind the dashboard sit service levels, the explicit statements of what "healthy" means for each integration. A service level objective might be that ninety-nine percent of orders sync within five minutes, or that the nightly reconciliation completes before the business day starts. These objectives are what make monitoring meaningful, because they convert vague notions of "working" into measurable targets you can track and alert against. Without them, every judgement about whether an integration is performing acceptably is a matter of opinion. With them, it is a matter of fact, and the dashboard becomes a scoreboard against a target rather than a wall of numbers open to interpretation.

8. Audit and compliance logging

Operational logging exists to help you run and troubleshoot the system. Audit logging exists to prove, after the fact, what the system did. They overlap but they are not the same, and conflating them causes problems in both directions. Operational logs can be verbose, short-lived and sampled; audit logs must be complete, tamper-evident and retained for as long as the relevant regulation or contract requires. A debug log you can rotate away after a week. An audit record of who approved a payment or which records were synchronised may need to survive for years and stand up to scrutiny.

For integrations that move financial data, personal data or anything else subject to regulation, audit logging is not optional. You need to be able to demonstrate what data crossed the boundary, when, in which direction, and on whose authority. In a regulated environment that evidence is the difference between a clean audit and a finding. When integrations carry financial postings between systems, for example the flows described in the Business Central APIs and integrations guide, the audit trail is part of the financial controls, not an IT afterthought, and auditors will ask to see it.

This is also where monitoring meets security, and the two reinforce each other. The same detailed logging that helps you diagnose a failed transaction also lets you detect a suspicious one: an unusual volume of extractions, access from an unexpected source, a spike in authentication failures at the integration boundary. Log with security in mind, protect the logs themselves from tampering, and be careful never to write secrets or unmasked sensitive data into log lines where they do not belong, a mistake that turns a helpful log into a liability. The overlap with the wider integration security practices is deliberate: observability and security draw on the same instrumentation, and a well-logged integration is easier to both operate and defend.

9. References

The frameworks and concepts this guide draws on are established industry practice rather than any one vendor's product. For readers who want to go deeper into the foundations:

• The three pillars of observability: the widely used framing that healthy observability rests on logs (discrete event records), metrics (aggregated numeric time series) and traces (end-to-end records of a single request across systems). Any serious observability text or platform documentation covers these three and how they complement one another.
• Distributed tracing and context propagation: open standards define how a trace identifier and per-hop span identifiers are carried in message and request headers so tracing tools can reconstruct an end-to-end flow automatically. These standards are what make correlation across independently built systems interoperable.
• Service level objectives and error budgets: the site-reliability practice of defining measurable objectives for a service and managing to them, which turns "is it healthy" from opinion into a tracked target.
• Dead-letter queue and messaging patterns: established enterprise integration patterns for handling messages that cannot be processed, containing failure without losing the transaction.

The point of naming these generically rather than pinning them to a specific product is that they are portable. Whatever tools you standardise on, the concepts are the same, and a team that understands logs, metrics and traces, correlation IDs, dead-letter handling and service levels can apply them on any platform.

Final thoughts

Observability is the difference between an integration estate you operate with confidence and one you cross your fingers over. The engineering is not hard. Emit a correlation ID at the edge and propagate it everywhere. Log discrete events with enough context to diagnose them. Track the handful of metrics that reveal health: throughput, latency, error rate, queue depth, retries and dead-letter count. Watch the dead-letter queue as if every message in it were a customer complaint, because it is. Alert only on what needs a human right now, and ruthlessly delete the alerts that cry wolf. Put a clean dashboard on top and measure it against explicit service levels. Keep an audit trail that would satisfy a regulator. None of these steps is difficult in isolation, and together they turn a black box into a system you can trust.

The organisations that get burned by integrations are not the ones with the most complex flows. They are the ones that shipped their flows blind, discovered the blind spots the hard way, and then had to reconstruct what happened from fragments. Build observability in from the start, as part of the integration design itself, and you never have that conversation. An integration you can see is an integration you can run. An integration you cannot see is, as the lead said, an incident waiting to happen. Make yours visible before it makes you find out.

Related reading: Enterprise system integration explained, Integration error handling best practices, Integration security best practices, How to design integration architecture, Business Central APIs and integrations.