BMS Ingestion Monitoring: Knowing Your Data Pipeline Is Healthy

Building data pipelines are monitored, if at all, by checking that the components are running. The gateway process is up, the broker is reachable, the database accepts connections. All three can be true while nothing has been collected since Thursday, because a pipeline can be perfectly healthy and idle, and nothing in a conventional uptime check distinguishes the two.

The message up front: monitor flow, not liveness. The question is not whether the components are running but whether the expected volume of data arrived in the expected time with the expected lateness. That requires a declared expectation per stage, a heartbeat the pipeline writes about itself, and alerting on absence rather than on error. It also requires separating pipeline health from data quality, because they have different owners and conflating them means neither gets fixed.

1. Why uptime monitoring misses this

Uptime answers whether a thing is responding. A pipeline can respond and do nothing, for reasons that are all common.

• Subscriptions lapsed and nothing is pushing, while every component is up. The mechanism is in COV subscription recovery.
• A collector lost its read position and is polling a device that has nothing new, or re-reading the same range.
• A rule or filter silently dropping everything after a configuration change that was syntactically valid.
• Writes failing on a constraint and being discarded rather than retried, which is a plausible outcome of adding the uniqueness key discussed in handling duplicate sensor readings without handling the rejection.
• The building side unreachable while the pipeline side is entirely well, which is the case covered in the connectivity loss playbook.

In every one of those the components are green. The only observable difference is that data stopped arriving, which means volume and recency are the primary signals and everything else is secondary.

2. The signals worth monitoring

The last row is the one most often missing and the one that catches the slow decay. Aggregate volume can look entirely normal while a steadily growing set of individual points has gone quiet, because the busy points dominate the total.

3. Expectation per stage, or the signals mean nothing

Volume and recency alarms need something to compare against, and that comparison has to be declared rather than inferred.

• Expected readings per period per source, derived from point counts and declared intervals rather than from last week's actuals. Deriving the expectation from history means a gradual decline becomes the new normal and never alarms.
• Expected interval per point, which is the field in the register described in the BMS point dictionary. Without it there is no basis for calling a point silent.
• Normal lateness per collection mechanism, measured once and recorded, because a polled point and a trend-log point have legitimately different profiles, per late and out-of-order readings.
• Allowance for genuine variation. Change-driven collection is naturally quieter overnight, and an alarm that fires every night at two will be muted within a week.
• A recalculation trigger. Expectations change when points are added, intervals altered or mechanisms switched, and a stale expectation produces either false alarms or silence.

4. Keep pipeline health and data quality apart

They are easy to merge into one dashboard and should not be, because they route differently.

Pipeline health is about whether data moved: heartbeats, volume, lateness, backlog, write failures. The owner is whoever runs the pipeline, and the fix is technical.

Data quality is about whether what arrived is trustworthy: coverage, completeness, freshness, validity, as set out in BMS data completeness KPIs. The owners are instrumentation, controls and the mapping, and the fixes are physical or configuration work.

The distinction matters because a combined alert is actionable by nobody. "Data quality degraded" sent to a shared inbox gets read by people who each conclude it is somebody else's. Separate signals with named owners get fixed.

5. Alerting that survives contact with an on-call rota

• Alert on the condition, not on every occurrence. One alert saying collection from a site has stopped, not one per silent point. A hundred alerts for one cause is the fastest route to muting.
• Group by cause, not by symptom. Forty points silent behind one controller is one alert about the controller, which is the cascade-grouping principle in alert deduplication between BMS and CMMS.
• Only genuine urgency notifies out of hours. A collection gap overnight that will be visible in the morning queue is not worth waking somebody for, unless the data drives something time-critical. The severity reasoning is in alarm severity and escalation design.
• Alert on trend as well as threshold. A backlog growing steadily is a problem before it crosses any limit, and the trend gives you time the threshold does not.
• Review the alerts that fired and were ignored. Those are either badly tuned or genuinely unimportant, and both conclusions are useful.

6. What the operational view should show

• Last successful collection per source, as a single list sorted by age. The most useful screen in the whole set, and the one that answers the common question immediately.
• Arrival volume against expectation, as a trend rather than a number, because the shape carries the information.
• Count of points currently silent beyond their interval, with the list one click away so it is a work queue rather than a statistic.
• Backlog depth and lateness, trended together.
• Current heartbeat status, including when it was last seen, so the monitoring is itself visibly monitored.

Keep this separate from the facilities-facing dashboards. The audience is whoever maintains the pipeline, and mixing it with building performance information means neither group reads it.

7. Starting from nothing

• Add the heartbeat first. Smallest change, catches the most damaging failure, needs no expectations to be declared.
• Then last-collection-per-source. Also needs no expectation, and it immediately reveals sources that stopped some time ago.
• Then arrival volume, with the expectation derived from the register, which is where the point dictionary becomes a prerequisite rather than a nice-to-have.
• Then per-point silence, which is the highest-value and needs declared intervals to exist.
• Expect the first run to find existing problems. Sources that have been quiet for months are a normal first finding, and finding them is the point rather than an embarrassment.

The idea to walk away with

Pipelines fail by going quiet, and quiet is invisible to monitoring built around errors and uptime. The whole discipline reduces to watching for the expected thing that did not arrive, which means declaring what is expected and having the pipeline report on itself from outside itself.

Then keep pipeline health separate from data quality, group alerts by cause, and put last-successful-collection-per-source somewhere people look. That one list answers most questions before they become investigations.

Final thoughts

The diagnostic that costs one query: for every source feeding your platform, when did it last deliver anything. Sort ascending.

On estates that have never checked, the top of that list is usually older than anyone would guess, and at least one entry turns out to be a source that stopped during a change nobody connected to it. That list is both the finding and the argument for building the rest.

Related reading: BMS data completeness KPIs, Connectivity loss playbook, COV subscription recovery, Late and out-of-order readings, BMS point dictionary, Store-and-forward buffering.