Planning a BMS Anomaly Detection Pilot That Can Produce an Answer

Anomaly detection pilots on building data have a characteristic ending. Three months in, the system has produced a large number of findings, a handful of them led somewhere interesting, nobody can say what proportion that is because nobody defined what success looked like, and the decision about whether to proceed is settled by whoever is most enthusiastic. The pilot was not badly executed. It was badly scoped, and it was badly scoped in a way that made it incapable of producing an answer.

The message up front: write down the question, the points, the period, the people, the success criteria and the stop conditions before you start, and make the success criteria something a negative result can fail. Scope narrow, on one equipment class and a small set of well-measured points. Route output to a named reviewer rather than to an operational queue. Budget the reviewer's time as the main cost, because it is. And decide in advance what you will do with a positive result, because a pilot with no route to production is an expensive way to learn something you cannot use.

1. Start from a question, not a capability

The weakest pilots begin with a tool and look for somewhere to point it. The strongest begin with something the organisation already wanted to know and could not find out.

A usable question is specific about the equipment, the behaviour and the consequence. Something like: are there conditions on our air handling units that we are not currently detecting and that would be worth attending to before they become callouts. That is answerable, it bounds the scope, and it makes a negative answer meaningful, because finding none is a genuine and valuable result on that equipment class.

An unusable question is whether machine learning can add value to our building data. Nothing can fail that, which means nothing can pass it either.

2. Scope: narrow beats representative

The instinct is to cover a representative slice of the estate so the result generalises. In practice breadth is what sinks pilots, because every additional equipment class brings its own data problems and its own reviewers.

• One equipment class. It keeps the review consistent, lets one person build intuition, and makes the findings comparable to each other.
• A small set of points per unit, chosen for measurement quality rather than for interest, using the checks in data readiness.
• Enough units to see variation between them, since a difference between nominally identical units is often the most informative output and needs more than two to appear.
• One site, unless the question is explicitly about multi-site comparison. A second site doubles the data problems and the stakeholders.
• Assets whose mapping is verified, or findings arrive with nowhere to route them, per point-to-asset mapping.

3. What the plan has to contain

4. Disposition categories, fixed in advance

This is the element most often skipped and it is what makes the pilot measurable. Every finding gets exactly one disposition from a short list, recorded by the reviewer at the time.

A workable set distinguishes a real condition worth acting on, a real condition already known, a data or collection artefact, a known operational event such as a shutdown, a correct observation of no consequence, and genuinely unresolvable. That last category matters and is usually omitted: some findings cannot be settled either way, and forcing them into a verdict corrupts the count.

The data artefact category is the one to watch during the pilot rather than at the end. A rising share there is a signal about the pipeline, not about the method, and it should trigger the ingestion and completeness checks from ingestion monitoring before more review time is spent. The deeper question of who decides what counts as real is covered in labelling BMS anomalies.

5. Where the output goes during the pilot

A pilot's output is not operational, and treating it as operational is the fastest way to lose the goodwill the pilot depends on.

• A separate queue, not the alert stream. Pilot findings competing with real alarms will either be ignored or will displace something that mattered.
• No automatic work order creation. The reviewer raises work where it is warranted, which keeps the maintenance record clean and keeps the classification in alarm code mapping meaningful.
• No notifications to anybody outside the pilot. A finding reaching an operations manager before it has been reviewed creates a conversation the pilot cannot support.
• A record that survives. Findings, dispositions and timestamps need to be queryable at the end, which means a table rather than a chat thread.
• Visibility for the controls team. They will frequently explain a finding in one sentence, and excluding them wastes reviewer hours.

6. The ways pilots fail, and the cheap protections

• Review backlog. The commonest ending. Protected against by setting scope from review capacity and by tuning sensitivity down rather than letting volume accumulate.
• Data problems discovered late. Protected against by running readiness checks first, which converts a pilot failure into a data finding.
• Moving goalposts. Protected against by writing success criteria down and circulating them before any output exists.
• The champion leaves. Common and fatal. Protected against by having the plan and the record owned somewhere institutional rather than by one person.
• A single dramatic early find. Genuinely a risk: one good catch in week two tends to end the evaluation and produce a decision based on an anecdote. Keep running to the agreed period.
• No route to production. Protected against by deciding in advance who would own a live service and where its output would go, which is the question in monitoring drift.

The idea to walk away with

A pilot's value comes from its ability to produce a clear negative, and that ability is designed in before it starts, through a specific question, a narrow scope, fixed disposition categories and success criteria that can fail.

Set the scope from the review hours you actually have, keep the output in its own queue with no automatic work creation, run to the agreed period even after a good early find, and decide in advance what a positive result leads to and who owns it.

Final thoughts

Before committing to anything, write the question on one line and ask whether a negative answer would change a decision somebody is going to make.

If it would, the pilot is worth running and the rest of the plan follows from it. If it would not, what is being proposed is a demonstration rather than an evaluation, which is a legitimate thing to want but should be budgeted and described as one.

Related reading: Data readiness, Pilot evaluation, Thresholds versus statistical methods, Labelling BMS anomalies, AHU telemetry quality checklist, Asset criticality.