BMS Connectivity Loss: Maintenance Escalation Playbook

A plant fault announces itself. A communications fault does the opposite: it removes the mechanism that would have announced anything. The building keeps running on whatever the controllers were last told to do, the front end keeps displaying the last values it received, and nothing turns red because nothing is being reported. In a well-run estate this is caught within minutes. In most estates it is caught when somebody notices a number has not changed since the weekend, or when a fault that was reported by nobody turns into a complaint.

The message up front: treat loss of visibility as an event in its own right, with its own alarm, its own severity, its own work order type and its own escalation path, which usually runs to a different party than plant faults do. Detect it by absence rather than by alarm, because an absent system raises nothing. Establish the blast radius before escalating, since one device and one site are very different conversations. And decide in advance what happens to the plant while you are blind, because that decision made under pressure is always worse.

1. Why this is not a plant fault

Three differences change how it should be handled, and conflating it with ordinary reactive work is why it usually gets handled badly.

• The plant is probably fine. Controllers hold their logic and keep running. What you have lost is supervision, not control, provided the architecture puts control in the controller rather than in the head end. Where schedules or interlocks live only at the supervisory layer, that assumption fails, which is one of the questions worth asking at selection time as discussed in BMS head-end platforms compared.
• The responsible party is usually different. A plant fault goes to the mechanical team. A communications fault could be the controls contractor, the network team, a cabling fault, a power supply, or a change somebody made to a firewall. The first task is determining which, and the default routing to a plant technician wastes a visit.
• The risk is cumulative and invisible. Every hour offline is an hour in which any condition can arise unreported. The cost is not the outage, it is whatever happened during it and was never seen.

2. The silent failure modes

Several things can break in ways that present as normal, and knowing them tells you what to monitor.

• Last known value persistence. Many systems hold the last received value indefinitely rather than marking it unknown. A space temperature of 22.4 that has not changed since Friday looks entirely reasonable. The same failure in a different guise is described in BMS points lists and field devices.
• Partial loss. One trunk of several, or one controller among forty. The site is up, the dashboard is up, and a slice of the building is unreported. Much harder to notice than a total outage and much more common.
• One-way loss. Reads still work, writes silently fail, so setpoints and schedules appear to be sent and are not applied. Rare and extremely confusing when it happens.
• Expired subscriptions. Where data arrives by subscription rather than polling, a subscription that lapses and is not renewed produces a connection that is technically healthy and delivering nothing new. The lifetime and renewal mechanics are covered in BACnet objects, device instances and addressing, along with the related case of a foreign device registration that stops being renewed.
• The integration, not the building. The building system is fine and the link to the maintenance system has stopped. Alarms are being raised and nobody is receiving them, which means the loss is invisible from both ends unless something is explicitly watching the link.

3. Detecting absence

You cannot wait for an alarm from a system that is not talking. Detection has to be positive: something must notice that something expected did not arrive.

Use at least staleness checks and an integration heartbeat. Device status alone is the most commonly implemented and the weakest, because the failure mode that hurts most is a device that is present and no longer telling you anything true.

4. Establishing the blast radius first

Before escalating, answer one question: how much is gone. The answer determines who you call and how urgently, and getting it wrong in either direction is costly.

• One device. Usually a local fault: power, a cable, a failed controller. Routine, unless the device serves something critical.
• One trunk or one network segment. Points at the shared infrastructure: a router, a repeater, a cable run, a termination. Often a single physical cause.
• One subnet. Frequently an IT change, and on building networks that regularly means a firewall rule or a switch configuration. The broadcast-forwarding mechanisms that break first in this scenario are described in BACnet objects, device instances and addressing.
• One site. Wide-area link, site power, or the site server. On a multi-site estate this is the scenario the architecture should already have accounted for.
• Everything. Head end, central server, or the integration layer. Check whether the plant is actually affected before treating it as an emergency; usually it is not.

Answering this quickly requires knowing which devices sit behind which infrastructure, which is a topology record rather than an asset register. Its absence is the most common reason a connectivity incident takes hours to attribute, and it is cheap to produce once and maintain.

5. The escalation playbook

Written in advance, on one page, available to whoever is on duty at two in the morning.

• Step one, confirm it is real. Distinguish a monitoring fault from a genuine loss. If one path says the device is gone and another says it is responding, the monitoring is wrong and that is a different job.
• Step two, establish the radius using section 4. Record it, because it is the single most useful fact for everyone who gets involved later.
• Step three, check the obvious shared causes before calling anyone: recent planned work, a known change window, site power, and whether anything else on the same infrastructure is also affected.
• Step four, route by radius. A device goes to controls. A subnet goes to IT and controls together, because it is frequently a network change and arguing about which it is wastes the first hour. A site goes to whoever owns the wide-area link, with controls informed.
• Step five, assess the plant exposure in parallel rather than afterwards: what is behind the loss, how critical is it, and what is the consequence of it running unobserved. This is where asset criticality classification earns its place.
• Step six, decide on compensating measures, covered below.

Route by radius rather than by symptom, and name the roles rather than individuals, for the same reasons set out in alarm severity and escalation design. The severity of a connectivity loss is not fixed either: losing supervision of a critical plant room is a different event from losing a lighting controller in a store room, and the scheme should let those differ.

6. Running blind: the compensating measures

The part most playbooks omit, and the part that matters while the fault is being fixed.

• Decide the threshold for physical rounds. At some level of criticality and some duration, somebody should go and look at the plant, because the normal mechanism for noticing problems is absent. Agree that trigger in advance rather than debating it during the incident.
• Know which protections are independent of the BMS. Local safety devices, mechanical protection and standalone controls generally keep working when supervision does not. Knowing which ones apply is an engineering question to answer before you need it, not during.
• Consider whether to suspend automatic changes. If scheduled changes are issued from a supervisory layer that is partially reachable, partial application can leave plant in an inconsistent state. Often the safer course is to hold changes until visibility returns.
• Tell the people who will be affected. Occupant complaints during a blind period will not be preceded by an alarm, so the helpdesk is temporarily your detection mechanism and should know it.
• Log the blind window precisely. Start and end. It bounds the period during which any later problem may have gone unreported, and it is the first question asked afterwards.

7. Recording it properly

Connectivity incidents are routinely mis-recorded, which is why most estates cannot say how often they happen.

• Give them their own work order type. Not a plant fault and not a general IT ticket. Without a distinct type they cannot be counted, and the classification question is the one addressed in CMMS work order types.
• Raise one record for the incident, not one per affected device. Forty points going quiet together is one event. Treating communications loss as a cascade parent is exactly the case discussed in alert deduplication between BMS and CMMS.
• Record the radius, the blind window and the attributed cause as structured fields. The attributed cause is what eventually shows you that most of your outages are one firewall or one ageing switch.
• Record whether data was recovered on reconnection, per section 8.
• Keep the audit trail intact across the gap, following the alert-to-work-order audit trail, so that the absence of events during the window is explained rather than looking like a quiet period.

8. Recovery, and the backfill question

When the link returns, three things need checking and usually only the first one is.

• Is everything actually back? Partial recovery is common, particularly where some devices reconnect automatically and others need a restart. Compare the device list against the pre-incident baseline rather than glancing at the dashboard.
• Did anything buffer, and did it arrive? Some controllers and gateways store readings locally during an outage and forward them on reconnection. If yours do, those records will arrive with old timestamps, and whether your systems handle out-of-order arrivals correctly determines whether the history has a hole or is complete.
• What happened during the window? Review any buffered data for conditions that occurred unreported. This is the step that occasionally finds something significant and is almost always skipped because the incident feels closed once the link is up.
• Did subscriptions re-establish? A reconnected device whose subscriptions did not renew is connected and silent, which is the failure this whole article is about, now hiding behind an apparently successful recovery. Verify explicitly.
• Were any alarms missed at the boundary? Conditions that raised and cleared entirely within the window may never appear anywhere unless buffered.

9. Reducing the frequency

• Attribute every incident to a cause and count the causes. Most estates find a small number of recurring sources, and fixing one of them removes a large share of the incidents.
• Get building control networks into the IT change process. A substantial proportion of subnet-level losses are caused by changes made without anyone realising the building depended on them. This is an organisational fix, not a technical one, and it belongs in the same conversation as BMS cybersecurity for connected buildings.
• Know what your service contract covers. Response to a communications fault is often outside a maintenance contract scoped around plant, and that gap is discovered during the first out-of-hours incident. Check it against your BMS service contract before you need it.
• Treat ageing network infrastructure as an asset with a lifecycle, alongside the controls themselves, per BMS maintenance, servicing and lifecycle.
• Test the detection periodically. Deliberately disconnect something during a planned window and confirm the alarm, the routing and the staleness display all behave. Detection that has never been tested is detection you are assuming.

10. What to measure

• Total blind time per month, weighted by the criticality of what was unobserved. The headline number, and the one that funds improvement.
• Time to detect. From the actual loss to somebody knowing. Often much longer than people assume, and the easiest thing to improve.
• Time to attribute. From detection to knowing whose problem it is. Usually the largest component, and the one the topology record fixes.
• Incidents by attributed cause. The list that tells you what to actually fix.
• Devices with the most incidents. Repeat offenders are usually a power, cabling or environmental problem at one location.
• Recovery completeness. How often full service was restored on the first attempt, and how often buffered data arrived intact.

The idea to walk away with

Every monitoring system has a blind spot in the same place: it cannot report its own absence. Closing that gap takes positive detection, something watching for the expected thing that did not arrive, plus an interface that shows age rather than a stale number.

After that it is ordinary incident management, done properly: establish the radius before escalating, route by radius rather than symptom, decide the compensating measures in advance, record it as its own event type, and verify on recovery that everything really came back, including the subscriptions nobody thinks to check.

Final thoughts

The test that tells you where you stand takes five minutes. Pick a controller, have it disconnected during a planned window, and time three things: how long until something alarms, how long until a human knows, and how long until the right party is engaged.

A surprising number of estates discover that the first number is never, because nothing was watching for absence, and that the graphic carried on showing yesterday's temperature throughout. That is a worthwhile thing to find out deliberately on a Tuesday afternoon rather than accidentally after a weekend.

Related reading: When a fault clears before the technician arrives, Alarm severity and escalation design, BACnet objects, device instances and addressing, BMS cybersecurity for connected buildings, BMS service contracts, BMS points lists and field devices.