A year into a monitoring programme somebody asks a reasonable question. How long did that plant item run outside its normal band on the night of the incident, and how many times did it cycle? The data is all there, in the sense that every day of the period is present in the store. It just cannot answer the question, because what survived is an hourly mean, and an hourly mean of a value that swung wildly for twenty minutes looks almost exactly like an hour in which nothing happened.
The message up front: aggregation is lossy in a specific and predictable way, and the loss falls entirely on questions about variability, duration and sequence. Choose the aggregate functions from the questions you expect rather than from habit, keep minimum, maximum and a sample count alongside any mean, never aggregate a counter the way you aggregate a measurement, and record which function produced a stored value so that nobody downstream has to guess.
1. What aggregation actually removes
It is worth being precise about the loss, because stated generally it sounds tolerable and stated specifically it often is not.
- Excursion magnitude. A mean flattens peaks. A value that exceeded a band briefly leaves no trace in an average over any interval much longer than the excursion.
- Duration above or below a threshold. This cannot be recovered from a mean at all. Two very different hours produce the same average, and the distinction is exactly what a compliance or comfort question turns on.
- Cycle counts and transitions. How many times something started, stopped or changed state within the interval is gone the moment the interval is summarised, unless the count was computed before the rollup.
- Sequence between points. Which of two things happened first is unanswerable once both are bucketed into the same interval, which matters whenever you are trying to establish cause.
- The evidence of absence. A mean computed over three readings and a mean computed over ninety-six look identical unless the count travelled with them, which is the link to the completeness measures in data completeness KPIs.
What aggregation preserves well is level and trend over time. That is genuinely most of what a reporting layer needs, which is why the practice is sound. The failure is in applying it to points whose value lies in their variability.
2. Choosing the function from the question
| Point character | Keep | Because |
|---|---|---|
| Slowly varying measurement space temperature, humidity |
Mean, min, max, count | The mean carries the level, the extremes carry the comfort or compliance question, the count qualifies both |
| Fast-moving measurement flow, pressure, differential |
Mean, min, max, count, and a longer raw window | Variability is the signal, and it is unrecoverable once summarised |
| Cumulative counter energy, runtime hours, pulses |
Last value in interval, plus the interval difference | Averaging a counter is meaningless. The difference is the quantity consumed, and resets break it, as in meter resets and rollover |
| Binary or state run status, mode, position |
Time in each state, transition count | A mean of a binary is a duty fraction, which is useful, but it loses the cycle count entirely |
| Setpoint or command | Last value, change count | What matters is what it was set to and when it was changed, not its average |
The test: can the aggregate still answer the question that justified collecting the point?
Every point was collected for a reason, even if the reason is now undocumented. Before setting a rollup policy, state that reason as a question and check that the aggregate you intend to keep can still answer it. A point collected to demonstrate that a space stayed within a band needs the extremes and the time outside the band, not the mean. A point collected to apportion consumption needs the interval difference, not the average. A point collected because it came with the controller and nobody knows why is a candidate for the shortest retention you have, and identifying those is a useful side effect of the exercise.
3. The unit trap in aggregation
Aggregation and unit semantics interact badly, and the result is one of the more common silent errors in building analytics.
A rate and a quantity aggregate differently. A rate, something measured per unit time, averages correctly and sums incorrectly. A quantity, something accumulated over the interval, sums correctly and averages into a figure with no physical meaning. If the stored metadata does not distinguish the two, a generic rollup job will apply the same function to both and produce one correct series and one that is quietly wrong. The same confusion appears earlier in the chain and is set out in sensor units and scaling and in validating meter units.
The practical defence is to refuse to aggregate any point whose kind is not recorded. A rollup that skips unclassified points and reports them as a work list is far better than one that guesses.
4. Labelling what you stored
The second structural requirement is that an aggregated value must be distinguishable from a raw one.
- Record the function and the interval with the value, not only in the job configuration. Configuration changes and history does not.
- Record the sample count. This is what lets a consumer know whether an aggregate is trustworthy, and it is the cheapest thing in the row.
- Keep the interval boundary convention explicit. Whether a bucket is labelled by its start or its end, and in which time zone, decides whether two systems agree about what a given hour contains. The timestamp mechanics are in timestamps and clock sync.
- Never overwrite a raw series in place with its aggregate. Write the aggregate to its own series, so a mistake in the rollup is recoverable while the raw tier still exists.
5. Aggregates and data that arrives after the fact
Rollups are usually computed on a schedule, which assumes the interval is complete when the job runs. On building telemetry that assumption fails routinely, because gateways buffer during outages and release when the link returns.
The result is an aggregate computed over partial data and then never corrected, which is the most durable kind of error: it looks finished. Either delay the rollup long enough to cover normal buffering, or make it recomputable for any interval and recompute when late data lands, and record the recomputation so a number that changed can be explained. The storage behaviour this depends on is covered in late and out-of-order readings, and the buffering that causes it in gateway store-and-forward.
6. Writing the policy down
What makes this work in practice is a short, boring document that says, per class of point, what resolution is held for how long and which functions survive the step down. It belongs alongside the retention policy rather than inside a scheduler configuration, because the people who need to know are not the people who administer the job.
Two further things are worth putting in it. First, which analyses the organisation accepts it will not be able to run on data older than the raw window, stated plainly, because that is the decision being taken and it is better taken deliberately. Second, the route by which somebody can request a longer raw window for a subset of points, with the cost attached, from storage sizing. The wider retention framing sits in the data retention policy.
Limits
No sampling intervals, retention durations or resolution recommendations are given here, because the adequate resolution for a given purpose is a function of the plant, the question and the regulatory context, and a published figure would be read as a standard it cannot be. Nor does anything here assert what sampling rate is sufficient to characterise any particular item of equipment or to detect any particular fault; that is an engineering judgement requiring validation against the plant in question, and it is outside what I can responsibly state. This is written from the data architecture side, which is where I work.
The idea to walk away with
Aggregation trades storage for the ability to answer questions about variability, duration and sequence. That trade is often worth making and it is almost never made consciously, because it is configured once by somebody solving a capacity problem and then inherited by everybody solving analytical ones.
Derive the functions from the questions, keep extremes and a sample count beside every mean, treat counters and states as different kinds from measurements, label what you stored, and write the resulting policy where the people who will hit its limits can read it.
Final thoughts
A quick exercise exposes most of this. Take one real question from the last incident review, find the oldest data you would need to answer it, and try.
If it answers, your rollup policy is fit for the way the organisation actually uses its data. If it does not, you have found the gap while there is still raw data in the recent window to widen, which is a much better moment to find it than during the next investigation.
Disclosure
Alongside advisory work I also build a CMMS and CAFM platform, so I have a commercial interest in this category. Nothing above is a recommendation for it, and no vendor named here has paid for inclusion or had any editorial input. Weigh the analysis accordingly.
Setting a resolution policy?
Independent help deriving aggregate functions from the questions your estate actually asks, and writing the policy where its limits are visible. 22+ years across CMMS, CAFM, EAM and ERP implementations.
Book a conversationRelated reading: Storage sizing, Data retention policy, Sensor units and scaling, Late and out-of-order readings, Data completeness KPIs, Meter resets and rollover.
Muhammad Abbas
CMMS / CAFM Manager & Independent Advisor · 22+ years across enterprise CMMS, EAM, CAFM and ERP implementations in utilities, oil and gas, manufacturing, government and facility operations.
Work with me