Evaluating an Anomaly Detection Pilot on Building Data

Evaluating a detection pilot looks like a measurement exercise and is mostly an honesty exercise. The counts are easy. What is hard is stating plainly what the pilot could not tell you, resisting the pull of one memorable catch, and comparing the result against the only baseline that matters, which is what would have happened anyway.

The message up front: report the disposition breakdown as counts, separate the share that were data artefacts because that is a different finding entirely, state what you cannot know about missed conditions and do not estimate it, compare against what the existing alarms and the existing inspection routine would have caught, cost the review effort as the recurring price of the service, and write the recommendation so that a reader who disagrees can see which number they are disagreeing with.

1. The counts that matter

Start with the plain breakdown, in counts rather than percentages, over the agreed period and scope.

• Findings produced, and findings reviewed. If these differ, the gap is the first thing to report, because everything downstream is conditioned on the reviewed subset and a backlog biases it.
• Each disposition category, counted. The fixed set agreed at planning, as described in pilot planning, with no recategorisation after the fact.
• Findings that led to work. A work order raised, a setting changed, an inspection brought forward. This is the concrete output and it is usually a small number.
• Findings that were already known. Important and frequently overlooked. A correct detection of something operations had been discussing for a month is a validation of the method and not a benefit.
• Unresolvable findings. Report rather than redistribute. A large unresolvable share is itself a result, usually about explainability.

2. Separate the data artefacts, and read them differently

The share of findings that turned out to be collection gaps, flagged readings, unit changes or mapping events deserves its own treatment, because it answers a different question.

A high artefact share is not primarily a verdict on the detection method. It is a measurement of data quality, obtained the expensive way. It means the readiness work in data readiness was incomplete, and it is genuinely useful information, because those artefacts are real problems affecting every other use of the same data.

So the honest reporting is two findings rather than one: here is what the method found about the plant, and separately, here is what the pilot found out about our data. The second is often the more valuable of the two, and presenting it as a pilot failure wastes it. The distinction between a data-path anomaly and a plant one is the subject of communication versus equipment anomalies.

3. Compare against doing nothing differently

The implicit comparison in most pilot reports is against no detection at all, which nobody was proposing. The real alternative is the current arrangement.

Findings that only this method could have caught are the actual case. Findings a reviewed limit would have caught are an argument for reviewing limits, which is cheaper and should be done either way.

4. Cost it as a service, not as a project

A pilot's cost is usually presented as the licence or the build. The recurring cost is review time, and review time does not fall once the pilot ends.

• Reviewer hours per week, measured from the pilot rather than estimated, and projected onto the scope you would actually deploy. This is the number that decides feasibility.
• Data maintenance. Mapping upkeep, register changes, unit corrections, and the crosswalk discipline in point naming and cross-system joins.
• Model and baseline maintenance, which is a continuing obligation rather than a one-off, per monitoring drift.
• Pipeline monitoring, since a silent input failure produces a silent service failure, from ingestion monitoring.
• The cost of wasted attendances, counted from the pilot's own dispositions rather than assumed to be negligible.

Scaling matters here and is usually assumed to be favourable. Ten times the points does not mean ten times the findings if sensitivity is tuned, but it does mean more assets, more reviewers, more mappings and more regimes, and the review burden per site tends to be the binding constraint on how far this goes.

5. Writing the result so it survives being quoted

A pilot report gets summarised by somebody who did not read it, and the summary is what gets acted on. That is worth designing for.

• Lead with the counts and the period, so the scale is attached to every claim from the start.
• State the scope limits in the same breath as the result. One equipment class at one site over one season, said plainly, prevents the result being generalised past what it supports.
• Make the recommendation conditional and specific. Proceeding on a named equipment class with a named reviewer is actionable; proceeding in principle is not.
• Report the data findings separately, with their own recommendation, because they are worth acting on regardless of the detection decision.
• Include the dissent. If the controls engineer thinks most findings were explainable by configuration, record that, because it will surface later anyway and it is more credible coming from you.

6. What a negative result is worth

A pilot that concludes against proceeding is a success, and it is worth saying so explicitly in the report, because the alternative is an organisation that cannot run evaluations.

A clear negative delivers three things. It closes a question that would otherwise recur every budget cycle. It usually produces a concrete list of data problems worth fixing. And it tells you something about where the value actually is on your estate: if most findings were explainable by configuration, the finding is that your alarm configuration deserves attention, which is a cheaper and more certain return than the thing you were evaluating.

What makes a negative result usable rather than merely discouraging is that the criteria were set in advance. A pilot judged against criteria written before any output existed produces a conclusion nobody can relitigate, which is the whole reason for writing them down first.

The idea to walk away with

Evaluating a detection pilot is mostly a matter of reporting counts honestly and stating clearly what the pilot could not establish, particularly about conditions it may have missed.

Separate the data artefacts into their own finding, compare against reviewed limits and the existing routine rather than against no detection, cost the review effort as the recurring price, write the result so a quoted summary stays accurate, and treat a clear negative as the valuable outcome it is.

Final thoughts

One question settles most pilot reports. Of the findings that led to real work, how many would a reviewed alarm limit have caught first?

If the answer is few, you have a genuine case and the number to make it with. If the answer is most of them, the pilot has pointed at something cheaper and more certain than the system being evaluated, and following that is a better outcome than proceeding would have been.

Related reading: Pilot planning, Data readiness, Thresholds versus statistical methods, Labelling BMS anomalies, Monitoring drift, FM KPI framework.