OneLake for Operational and Sensor Data: A Design Guide

Most teams meet OneLake and immediately reach for the wrong mental model. They treat it like another storage account to fill up, copy data into, and forget. Operational and sensor data punishes that habit faster than almost any other workload: it arrives late, out of order, at high frequency, and it never stops. This guide is the design I actually use when I land historian archives and live sensor streams into OneLake, written as a set of decisions you can adopt rather than a feature tour.

OneLake is one logical lake, not another storage account

The single most useful thing to internalise is that OneLake is one lake for your whole tenant. There is exactly one, it is provisioned automatically, and every workspace and lakehouse lives inside it as a folder structure rather than as a separate account you stand up and wire together. It speaks the ADLS Gen2 APIs, so your existing tools still work, but the point is that you stop thinking in terms of "which storage account holds this".

That reframing changes how you handle data you already have. When a process historian has ten years of archived readings sitting in an ADLS Gen2 container, the instinct is to copy it into Fabric. Do not. OneLake shortcuts let you mount that container as if it were a native folder in your lakehouse. The bytes stay where they are, you pay no second copy, and queries read through the shortcut transparently. A shortcut is a pointer with a path, not a pipeline.

For the canonical description of both concepts, see the OneLake overview and the OneLake shortcuts documentation . Shortcut source support and the OneLake security model are both still expanding, so treat any specific source list you read today as a snapshot and recheck it quarterly.

A concrete zone layout for operational data

Medallion architecture (bronze, silver, gold) is a good spine, but it is too abstract for people staring at a historian export. Here is the layout I lay down, mapped to the four things operational data actually needs to become: a raw landing, a conformed asset master, a work-order fact, and a curated serving layer.

Raw sensor landing (bronze)

Everything lands here exactly as it arrived, untouched. Live streams write here, and the historian cold archive is mounted here by shortcut rather than copied. No schema enforcement, no dedup, no filtering. This zone is your audit trail and your replay source. If a downstream transform has a bug, you fix the code and reprocess from bronze; you never lose the original because you never overwrote it.

Conformed asset master (silver dimension)

Sensor rows are meaningless without the asset they describe. This zone holds the cleaned, deduplicated, slowly-changing asset dimension: tag-to-equipment mappings, asset hierarchy, criticality, location. It is small, it changes slowly, and it is the join key everything else depends on. Get this wrong and every report downstream is wrong in a way nobody notices until an audit.

Work-order fact (silver fact)

Operational data is not only sensor readings. Work orders, alarms, and events are facts that reference the asset dimension. Modelling them as a proper fact table, with the asset key and an event timestamp, lets you correlate a vibration spike against the maintenance that followed it. This is where sensor context and maintenance history finally meet.

Curated serving layer (gold)

The shaped, downsampled, business-ready tables that Power BI, notebooks, and the SQL endpoint read. Pre-aggregated rollups, KPI tables, and downsampled tag histories live here. Nothing in gold should require a heavy scan of bronze at query time. If a dashboard is slow, the fix is almost always a missing gold table, not a bigger capacity.

The folder and shortcut layout, drawn

Below is the physical shape of it. Notice that the historian cold archive is not inside OneLake at all; it stays in ADLS and appears in bronze through a shortcut. The arrow into bronze is a mount, not a copy.

The dashed terracotta line is the shortcut. Reads flow from the external archive into bronze on demand; no bytes move on a schedule. Everything downstream of bronze is native OneLake, written by your own transforms.

Partitioning and the small-file problem

Sensor data is a small-file factory. A stream writing every few seconds, partitioned naively, produces millions of tiny Parquet files, and query planners choke on file counts long before they choke on data volume. The partitioning scheme that survives contact with real historian volumes is date plus asset group: partition by ingest date first, then by a coarse asset grouping (plant, area, or equipment class), never by individual tag.

Partitioning by individual tag feels tidy and is a trap. Ten thousand tags becomes ten thousand partition directories per day, each holding a handful of rows. Group tags into a few dozen asset groups instead, and you keep partitions large enough to matter.

Compaction (rewriting many small files into fewer large ones) becomes urgent at the thresholds below. These are the rough numbers I watch, based on Delta tables read through the SQL and Spark engines:

The practical target is files in the 128 MB to 1 GB range and a few hundred rows per file at absolute minimum. If a partition is accumulating thousands of sub-megabyte files a day, no amount of capacity buys your way out; the writer is committing too often and needs bigger micro-batches or a downstream compaction job. Sizing the upstream historian correctly helps here too, which I cover in sizing a process historian.

Late and out-of-order data: the columns nobody writes down

Here is the detail that separates a design that works from one that quietly corrupts. Sensor data does not arrive in order. A gateway loses connectivity, buffers three hours of readings, and flushes them after readings from a different device have already landed. A field radio backfills yesterday at noon. If your silver layer assumes ingest order equals event order, you will compute wrong averages, miss gaps, and double-count backfilled points.

The fix is two timestamps, always kept as a pair, never collapsed into one:

• event_time → when the reading was actually taken at the sensor. This is the truth for all analytics and windowing.
• ingest_time → when it landed in bronze. This is how you detect lateness and drive incremental processing.

With both, a point that shows up three hours late is still placed on the correct hour by event_time, while ingest_time tells your incremental job that something changed for a window it already processed. You then need a deduplication key, because the same reading often arrives twice: once from the live stream and again from the historian backfill. I key on the tuple of (asset_id, tag_id, event_time) and keep the row with the latest ingest_time, on the assumption that a later arrival is a correction. That single dedup rule prevents the most common silent error in sensor lakes: the same second counted twice.

Why the asset dimension must be slowly-changing

The asset master is not static, even though it feels like it should be. Equipment gets recategorised, criticality is reassessed, a pump moves from one line to another, a tag is remapped to a replacement sensor. If you store the asset dimension as a simple overwrite table, every historical report is computed against today's attributes, not the attributes that were true when the reading was taken.

The consequence is subtle and serious: last year's downtime report silently rewrites itself. An asset that was "critical" during a Q3 outage but was downgraded to "standard" in Q1 this year will show, in any re-run of the Q3 report, as standard. The historical availability KPI shifts under your feet, and nobody made a data-entry error. The model did it.

The remedy is a slowly-changing dimension (Type 2): each asset attribute change creates a new versioned row with valid_from and valid_to timestamps, and facts join to the dimension version that was effective at the reading's event_time. It is more work, and it is not optional for anything that feeds compliance, downtime, or contractual availability numbers. If the asset dimension can be rewritten in place, so can your history. This is the same discipline that keeps historian-to-SCADA context correct, which I go deeper on in SCADA historian integration.

Do not dump every tag at one-second resolution

There is a strong temptation to land every tag at full one-second (or faster) resolution "just in case". Resist it. A few thousand tags at one-second resolution is billions of rows a month, most of which describe a value that did not change. You pay for the storage, you pay for every scan, and you pay in query latency, all to preserve noise. OneLake is not the right place to keep raw high-frequency signals that only a control engineer investigating a single event will ever look at.

Keep the full-resolution stream in the historian, where it belongs, and land a downsampled representation in OneLake driven by tag criticality. The policy I apply:

• Safety and regulatory tags → keep at native resolution or 1 second, no downsampling. These get subpoenaed.
• Critical process tags → 10-second aggregates plus exception logging on deadband, so real excursions survive but flatlines do not.
• Standard operational tags → 1-minute aggregates (min, max, avg, count).
• Slow or diagnostic tags → 5 to 15-minute aggregates.

The trick that keeps this honest is deadband or exception logging on the important tiers: you always record a point when a value moves beyond a threshold, even inside a downsampled window, so you never smooth away the spike that mattered. Full fidelity for the tags that need it, coarse rollups for the tags that do not, and the historian as the system of record for anything you deliberately did not keep.

A naming and medallion convention you can adopt

Conventions are worth more than cleverness, because everyone who touches the lake after you needs to guess correctly without asking. Here is the naming and medallion scheme I standardise on. Copy it as-is; consistency beats any individual choice below.

Two column names are non-negotiable in this scheme: event_time and ingest_time appear on every fact table, always UTC, always both present. If a future teammate sees only one, they should treat it as a bug to investigate, not a shortcut to copy.

Where this fits, and a word on independence

This design is one layer of a bigger picture. OneLake is the storage and modelling substrate; the ingestion, capacity planning, and governance decisions around it are a separate set of choices I lay out in the Microsoft Fabric adoption roadmap. Get the zone layout, the timestamp pair, the slowly-changing dimension, and the downsampling policy right, and the rest of Fabric has something clean to build on.

One honest caveat on currency. Fabric licensing, SKUs, and features move quickly, and both shortcut source support and the OneLake security model are still expanding. Everything here is verified as of 1 August 2026; the design principles are durable, but recheck the specific capability and source lists against Microsoft's documentation each quarter before you commit an architecture to them.

Independence note: I am not affiliated with Microsoft and receive no compensation for recommending Fabric or OneLake. The choices above reflect field experience, not a vendor relationship, and you should validate them against your own workload and licensing.