Every evaluation of building anomaly detection rests on somebody having decided, event by event, whether the thing reported was real. That decision is treated as a clerical step. It is in fact the hardest part of the exercise, because "real" is doing several jobs at once, two competent engineers will disagree on a meaningful share of cases, and the answer frequently depends on evidence that no longer exists by the time anybody asks.
The message up front: define what a label means before collecting any, and define it as two separate judgements rather than one, because whether a departure actually occurred and whether it was worth acting on are different questions with different answers. Allow an explicit uncertain label and keep it. Record who labelled, when, and on what evidence. Expect disagreement between labellers and measure it rather than averaging it away. And accept that a label applied weeks later is a weaker kind of evidence than one applied at the time.
1. Four things "real" can mean
When a reviewer says a finding was real, they could mean any of these, and they will mean different ones on different days.
- The data genuinely departed from its usual pattern. A statement about the series, checkable from the data alone, and the only one of the four that is purely objective.
- Something was actually different about the plant. A statement about the physical world, which requires evidence from outside the data.
- It was worth somebody's attention. A judgement about consequence and cost, dependent on criticality and on how busy the team is.
- It predicted something that later happened. Only knowable in hindsight, and the one people reach for when they want a strong claim.
These come apart constantly. A clear departure in the data caused by a known shutdown is real in the first sense and not in the others. A condition the reviewer judges unimportant is real in the first two senses and not the third. Collapsing all four into one yes or no is how evaluations become arguments, and it is why the recommendation below is to label at least the first two separately.
2. A labelling scheme that holds up
| Field | Values | Why it is separate |
|---|---|---|
| Departure occurred | Yes, no, cannot tell | Checkable from the data. Keeps the objective part objective |
| Cause category | Plant, instrument, data path, planned activity, unknown | The most useful single field, and it routes the work. Distinguishing the middle two is the subject of communication versus equipment anomalies |
| Action warranted | Yes, no, already in hand | A judgement about consequence, not about the data |
| Confidence | Confident, uncertain | Lets uncertain labels be excluded from a headline figure without discarding them |
| Evidence used | Data only, site visit, operator account, work order record | A label from a site visit is stronger evidence than one from a chart, and the difference should be visible |
| Labeller and timestamp | Person and time of labelling | Enables agreement measurement and shows the delay since the event |
Keep the uncertain label, and never force a verdict
There is strong pressure to eliminate the uncertain category, because it complicates the arithmetic and looks like indecision. Removing it does not remove the uncertainty; it hides it inside the yes and no counts, where it cannot be seen and cannot be reasoned about. A reviewer forced to choose will choose, and their choice will be influenced by what they expect the answer to be, which is exactly the bias the evaluation exists to avoid. Keep the category, report it, and compute headline figures on the confident subset while stating how large the uncertain share was. A large uncertain share is itself an important result: it usually means the output is not explainable enough to act on, which is a finding about the method rather than about the reviewer.
3. Label at the time, not at the end
The evidence that settles a label decays quickly, and faster than anybody expects.
A finding reviewed within a day or two can be checked: somebody can look at the plant, ask the operator what was happening, check whether works were underway. The same finding reviewed six weeks later can only be checked against the data and whatever happens to be recorded, which means the label becomes a judgement about a chart. Those two labels are not the same kind of evidence and should not be pooled without distinction, which is why the evidence field above exists.
This has a practical consequence for how a pilot runs. The fixed weekly review slot recommended in pilot planning is not administrative tidiness; it is what keeps labels close enough to the event to be worth anything. A backlog does not merely delay the evaluation, it degrades the quality of its ground truth.
4. Expect disagreement, and measure it
Two experienced people labelling the same set of findings will disagree on a non-trivial share, and this is normal rather than a sign that somebody is wrong.
- Double-label a subset deliberately. A sample labelled independently by two reviewers tells you how reproducible your ground truth is, which bounds how precise any conclusion drawn from it can be.
- Report the disagreement rate rather than resolving it silently. If two engineers disagree on a quarter of cases, no evaluation built on those labels can distinguish differences smaller than that, and saying so protects the conclusion.
- Look at where they disagree. Disagreement concentrated on the action-warranted field is a priorities conversation. Disagreement on the departure field means the output is unclear.
- Discuss the disagreements once, then sharpen the definitions. Most of the gap usually comes from different readings of the categories rather than different readings of the plant, and that is fixable.
- Do not resolve by seniority. It produces agreement without improving the labels and discourages the junior reviewer from recording genuine uncertainty.
5. The imbalance nobody plans for
Two statistical properties of this kind of labelled data are worth understanding, because they shape what the labels can support.
First, genuine conditions worth acting on are rare relative to flagged events, which means the labelled set is heavily skewed and a small number of positive labels carries most of the information. A handful of disputed labels can therefore move a conclusion substantially, which is another reason to record confidence and to report counts rather than rates.
Second, and more subtly, the labelled set is not a sample of conditions in the building. It is a sample of what the method flagged. Nothing in it describes the conditions that occurred and were never flagged, which is why the detection-rate caution in pilot evaluation applies. Labels can tell you how often the method was right when it spoke. They cannot tell you how often it stayed silent when it should not have.
6. Storing labels so they remain usable
Labels collected in a spreadsheet during a pilot have a short life, and their value is mostly in their reuse.
- Key them to the event, the point and the time window, with the stable point identifier from point naming and cross-system joins, not to a row number in a report.
- Record what the method was when the label was made. A label is a judgement about an output, and if the method changed, the pairing matters.
- Keep them append-only. A revised label is a new record referencing the old, so that a change of mind is visible rather than silent.
- Link them to the work that followed, through the asset event record described in linking faults to asset history, which is what turns a label into evidence.
- Include them in the backup inventory, since reconstructing a labelled set costs more than collecting it did, per backup and restore testing.
Limits
No acceptable agreement rate, label volume or class balance is given here, because what is adequate depends on the question being decided and any figure would be read as a standard. Nothing here is guidance on judging whether a particular plant condition was real or significant: that judgement belongs to engineers qualified on the equipment, and the argument above is precisely that the scheme should capture their uncertainty rather than compress it. Nothing here claims that a labelled departure indicates a developing failure or remaining life. No method, tool or vendor is recommended. This is written from the data and process side rather than as reliability or controls engineering.
The idea to walk away with
The ground truth in building anomaly detection is a set of human judgements, and the word "real" is doing at least four jobs that come apart in practice.
Split the label into whether a departure occurred, what caused it and whether action was warranted, keep an explicit uncertain value, label close to the event and record what evidence was available, double-label a subset to measure reproducibility, and store labels keyed to stable identifiers so they outlive the pilot.
Final thoughts
A quick exercise is unusually revealing. Take ten findings from any period, have two people label them independently against the scheme above, and compare.
Whatever the agreement turns out to be, you now know the precision ceiling on every conclusion your evaluation can reach, and you will almost certainly have found two or three category definitions that need sharpening. Both are cheaper to discover on ten findings than on three hundred.
Disclosure
Alongside advisory work I also build a CMMS and CAFM platform, so I have a commercial interest in this category. Nothing above is a recommendation for it, and no vendor named here has paid for inclusion or had any editorial input. Weigh the analysis accordingly.
No agreed definition of a real finding?
Independent help designing a labelling scheme that separates the objective question from the judgement, and measuring how reproducible your ground truth is. 22+ years across CMMS, CAFM, EAM and ERP implementations.
Book a conversationRelated reading: Technician feedback loops, Pilot evaluation, Pilot planning, Communication versus equipment anomalies, Linking faults to asset history, Failure codes.
Muhammad Abbas
CMMS / CAFM Manager & Independent Advisor · 22+ years across enterprise CMMS, EAM, CAFM and ERP implementations in utilities, oil and gas, manufacturing, government and facility operations.
Work with me