Store-and-Forward Buffering for BMS Gateways During Outages

Most building data pipelines are designed as if the network is always available, and then spend their lives discovering that it is not. A wide-area link drops for twenty minutes, a broker restarts, a firewall rule changes on a Friday afternoon. Without local buffering, every one of those events removes a piece of history permanently, and the piece removed is from exactly the period somebody will later want to examine.

The message up front: buffering only helps if three things are true together. The buffer has to be deep enough for the longest outage you actually expect, which means sizing it from evidence rather than from a default. Replayed readings must keep their original event times rather than being stamped on arrival. And the receiving system must accept data that arrives after newer data without rejecting it or putting it in the wrong order. Get any one of those wrong and the buffer gives you a false sense of resilience, which is worse than knowing you have none.

1. Where buffering can live

There are usually several candidate places, and they are not equivalent.

The gateway is where this matters most, because the link it sits behind is usually the least reliable part of the chain. The useful mental model is a chain of buffers, each covering failures downstream of itself, and the question to ask of a design is which failures are covered by nothing.

2. Sizing the buffer from evidence

Buffer depth is usually left at a default that nobody has compared against reality. The right way round is to measure first.

• Start from your actual outage history. How long have your real interruptions been, and what was the longest in the last year? Most estates have this in their incident records once somebody looks, and the distribution is usually a lot of short ones and one or two long ones.
• Size for the long tail, not the median. A buffer that covers the typical outage and not the unusual one will fail exactly when the loss matters most.
• Express the depth in time, not in records. "Ten thousand readings" means nothing operationally; "eight hours at current collection rates" is a number people can reason about, and it changes when the collection regime changes.
• Re-derive it when sampling rates change. Increasing collection frequency silently shortens the buffer in time terms, which is a connection almost nobody makes. The regime side is in BACnet polling schedules.
• Decide what happens when it fills. Discard oldest, discard newest, or stop collecting. All three are defensible and the default is rarely a decision. Discarding oldest loses the start of the outage, which is usually the most diagnostically interesting part.

3. Timestamps on replay, which is where it silently breaks

A buffered reading describes a moment during the outage. If the pipeline stamps it on arrival, it now describes the moment the link came back, and the entire outage period appears as a burst of identical-timed readings followed by a gap where the data actually belongs.

This is both the most common defect in store-and-forward implementations and the hardest to notice, because the data looks complete.

• Carry the event time through the whole chain, separately from publish and ingestion times, as set out in BACnet timestamp and clock synchronisation.
• Never let any stage substitute its own clock when an event time is present. Where one is genuinely absent, mark it absent rather than filling it in.
• Test it explicitly. Disconnect, wait, reconnect, and check that the recovered readings carry times from during the outage rather than from the reconnection. This is a two-minute test that most implementations fail, and it belongs in the acceptance test checklist.
• Watch for device clock errors surfacing here, because a buffered reading from a device with a wrong clock arrives with a wrong event time and is now much harder to spot than it would have been live.

4. What the receiving end has to tolerate

Replay delivers data that is late and out of order relative to what has already arrived. Several common storage designs handle that badly.

• Accept writes with event times earlier than existing rows. A store that assumes monotonic arrival will either reject them or append them in arrival order, producing a series that is wrong when read back by time.
• Be idempotent. Replays overlap: a gateway that is unsure what was delivered will usually re-send, which is correct behaviour. The receiver needs a key on point plus event time so duplicates collapse rather than accumulate.
• Recompute derived values for the affected window. Hourly rollups, daily totals and any aggregate covering the outage period were computed from incomplete data and are now wrong until recalculated. This is routinely forgotten, and it is why a day can show a correct detail series and an incorrect total.
• Re-evaluate, or deliberately not re-evaluate, alert rules. Conditions that occurred during the outage are now visible. Raising alerts retrospectively for conditions that have since cleared is usually wrong; recording them as occurrences is usually right, which is the disposition question in when a fault clears before a technician arrives.
• Mark the recovered window. Data that arrived by replay should be distinguishable from data that arrived live, so later analysis can see it.

5. What buffering does not solve

• It does not restore visibility during the outage. You are still blind in real time, and the escalation question is separate, covered in the connectivity loss playbook.
• It does not help with alarms that needed a response. An alarm recovered two hours late is history, not a notification.
• It does not cover loss of the buffering device itself unless the buffer is on persistent storage that survives a restart. An in-memory buffer evaporates on a power cut, which is one of the more likely causes of the outage in the first place.
• It does not cover the upstream side. If the gateway cannot reach the controllers, there is nothing to buffer, and only the controller's own trend log covers that case.
• It does not reduce the need to fix the underlying cause. A buffer that is regularly exercised is telling you something about the network.

6. Proving it works, and monitoring it

• The disconnect test. Interrupt the link for a defined period during commissioning, restore it, and verify three things: the data filled in, the event times are from the outage window, and no duplicates appeared.
• The overflow test. Interrupt for longer than the buffer holds and confirm the overflow alarms and the lost period is marked rather than silently absent.
• The power test. Remove power from the gateway mid-buffer and confirm whether the buffer survives. The answer determines whether your buffer is real.
• Monitor buffer depth continuously, not just as a pass or fail. A buffer that is routinely half full is a warning about the network, and it belongs in the measures in BMS ingestion monitoring.
• Count replays and their volume, trended. Rising replay volume is early evidence of a degrading link, usually well before anyone raises it as a fault.

The idea to walk away with

Store and forward converts an outage from permanent data loss into a delay, which is a large improvement and only if all three parts hold: a buffer sized from real outage history, event times preserved through the replay, and a receiving system that accepts late and out-of-order data idempotently.

Two of those three are properties of systems downstream of the gateway, which is why this is a pipeline design question rather than a gateway setting, and why it is so often half-implemented.

Final thoughts

If you want to know whether your buffering actually works, do not read the documentation. Disconnect something for ten minutes on a Tuesday, reconnect it, and look at what the event times say.

A surprising proportion of pipelines that are described as having store and forward turn out to replay the data with arrival timestamps, which means every outage in their history has been silently relocating readings to the wrong moment. Finding that deliberately costs ten minutes. Finding it during an investigation costs considerably more, because by then the affected history is already wrong and nobody can say how far back it goes.

Related reading: BACnet data logging, Timestamps and clock synchronisation, Connectivity loss playbook, BACnet to MQTT integration, BACnet acceptance test checklist, BMS data retention policy.