Monitoring Drift in a Live Building Anomaly Detection Service

A pilot ends well, the service goes live, and for a few months everybody is pleased. Then the findings tail off. The obvious reading is that the estate has improved, and it is the reading that gets reported. The other possibilities are that a collection path degraded, that somebody widened a sampling interval, that the baselines quietly absorbed a deteriorating condition as normal, or that a change in operating pattern put the whole estate outside what the service knows. Silence is consistent with all of these, and the service itself cannot tell you which.

The message up front: monitor three things separately. The inputs, so a collection failure is never mistaken for a quiet building. The output rate, with a declared expected range, so both silence and a flood are visible. And the agreement between findings and what they turned out to be, so a degradation in usefulness surfaces before anybody loses faith in the service. Version every model and baseline with its training window, because a result you cannot reconstruct is a result you cannot defend.

1. Three things drift, and they need telling apart

The word drift covers three distinct failures with three different owners and three different fixes.

• Input drift. The data arriving changes character: completeness falls, an interval widens, a unit changes, a point is remapped. Nothing is wrong with the model, and everything downstream of it is now unreliable. This is a data pipeline problem and it is by far the commonest.
• Behaviour drift. The building genuinely changed. New tenant, new occupancy pattern, a plant item replaced, a control strategy retuned. The baseline is now a description of a building that no longer exists, which is nobody's fault and still needs acting on.
• Performance drift. The findings are still arriving at a normal rate and they are less often useful. The hardest to detect, because it requires feedback to see at all, and the one that destroys trust.

Conflating them produces the wrong response. Retraining a model to fix what is actually a collection failure bakes the degraded data into the new normal, which is the worst available outcome because it is self-concealing.

2. Watch the inputs first

Input monitoring is the cheapest of the three and catches the most. It is also mostly already built if the data platform is monitored at all.

3. Declare an expected output rate

The output rate is the second thing to watch and the one most often left unmonitored, because nobody set an expectation for it.

What is needed is a stated range, derived from the pilot, outside which somebody looks. Both directions matter. A collapse towards zero is the silent failure above. A sudden flood is almost never the building and almost always a regime change, a configuration change or a data event, and it has a secondary cost: it exhausts the review capacity that the service depends on, which is the constraint identified in pilot planning.

It is worth tracking the rate per point group rather than only in aggregate, because the informative pattern is usually concentration. A flood confined to one site, one controller or one collection mechanism is a diagnosis in itself, in the same way that clustered completeness failures are.

4. Watching whether findings still mean anything

Performance drift cannot be seen from the data alone. It needs the disposition of findings over time, which means the feedback described in technician feedback loops has to keep working after the pilot ends, and that is where most services lose their grip.

• Track the disposition mix over time, in counts, using the same categories as the pilot so the comparison holds.
• Watch the data artefact share especially. A rising artefact share is input drift showing up in the output, and it is often the earliest visible symptom.
• Watch the already-known share. Rising means the service is increasingly telling operations what they already knew, which is a loss of value that no accuracy measure captures.
• Keep a labelled sample going. A small number of findings labelled properly each month, per labelling anomalies, is enough to see a trend and cheap enough to sustain.
• Review with the controls team. They will often identify a configuration change that explains a shift in one sentence, and they are the people who know what was retuned last month.

5. Retraining as a change, not a maintenance task

At some point a baseline needs updating, and the decision is more consequential than it appears, because it is a decision about what will count as normal.

The central hazard is retraining on a period in which something was already wrong. A slowly deteriorating condition included in the training window becomes the new expectation, and the service stops reporting it. This is the mechanism by which a detection service can quietly accept a degrading estate, and it is not detectable from inside the system.

Three protections are worth the effort. Treat the training window as an explicit choice that somebody reviews, with known problem periods excluded, using the shutdown and suppression records from planned shutdown suppression. Never retrain automatically on a rolling window without that review, however convenient it is. And keep the previous baseline so that a comparison is possible, since the useful question after a retrain is what stopped being flagged.

6. Versioning, so a past result can be explained

Six months after a finding, somebody will ask why the system flagged it, or more awkwardly why it did not flag something.

• Version the model or baseline and record which version produced each finding. Without this, a historical result cannot be reconstructed at all.
• Record the training window with each version, including what was excluded and why, because that is usually the actual answer to the question.
• Record configuration changes with dates, especially sensitivity adjustments, and keep them where a report author will see them rather than in a deployment log.
• Annotate trends at version boundaries. A step in finding rate at a retrain date will otherwise be read as a change in the building.
• Keep the inputs reproducible. This is the point where retention policy and detection meet: if the window a finding was computed on has been aggregated away, the result cannot be re-examined, which is a consideration for the retention policy.

The idea to walk away with

A live detection service fails silently, and its silence is indistinguishable from good news. The three things that drift, inputs, building behaviour and usefulness, have different causes and different fixes, and conflating them leads to retraining over degraded data.

Monitor the inputs and publish their completeness beside every output, declare an expected output range and watch both directions, keep a small labelled sample running to see usefulness change, and treat every retrain as a reviewed change with its training window recorded and its predecessor kept.

Final thoughts

If a detection service is already running, ask one question: for the last reporting period, what was the input completeness on the points it watches?

If the answer is on the report, the service is being run properly. If nobody knows, then every conclusion drawn from its output in recent months rests on an assumption nobody has checked, and adding that one figure is the cheapest reliability improvement available to the whole arrangement.

Related reading: Ingestion monitoring, Data readiness, Pilot evaluation, Technician feedback loops, Data completeness KPIs, Communication versus equipment anomalies.