Sizing a Process Historian: Tags, Compression and Retention

Every historian project I have been near starts the same way: someone quotes a terabyte-per-year number they read in a datasheet, panics, and either over-buys storage or under-scopes the tags that actually matter. Neither is necessary. Sizing a historian is a chain of four decisions you can cost out before you sign anything: how many tags, how fast each one scans, how hard you compress, and how long you keep it. Get those four right and the storage number falls out of the arithmetic. This is a how-to, so I will walk the chain in order and give you the tables to plug your own numbers into.

Step 1: Count your tags, and split them by class

A tag is a single measured point: one temperature, one motor run status, one energy meter register. The instinct is to count everything the control system exposes and historize all of it at the fastest rate the historian allows. That instinct is the single biggest cause of oversized, expensive, slow historians. The fix is to sort tags into classes by how the data behaves and how it will be used, then size each class differently.

Here is the classification I use as a starting point on a mixed building and process site. Your split will differ, but the principle holds: not all tags deserve a one-second scan.

The moment you draw this table for your own site, the fantasy of "everything at one second" dies. A 20,000-tag site is rarely 20,000 one-second tags. It is a few thousand fast control points and a long tail of status bits, meters and event data that barely move the storage needle. Classify first, then size.

Step 2: Turn scan rates into raw sample counts

Before compression enters the picture, work out the raw sample rate. This is the theoretical volume if you stored every scan. One tag at one second is 86,400 samples a day, roughly 31.5 million a year. The same tag at 15 minutes is 35,040 samples a year, nearly a thousand times fewer. Change-of-state and event tags produce whatever the process throws at them, usually a small fraction of a polled rate.

A single stored sample in most historians is on the order of a timestamp, a value, and a quality flag. Vendors quote anywhere from a handful of bytes to a couple of dozen bytes per event depending on data type and format, so treat per-sample size as a range, not a constant. I plan with roughly 4 to 12 bytes per compressed event and sanity-check against the vendor's own figures for the specific product and data types. The point of the raw number is not precision; it is to show you where the volume actually lives before compression touches it.

Step 3: Apply deadband and swinging-door compression

Historians do not usually store every scan. They store the scans that matter and reconstruct the rest by interpolation. Two mechanisms do the heavy lifting.

Deadband (exception) filtering:

The collector only passes a new value to the archive if it has moved more than a set amount, the deadband, from the last value it reported. A temperature sitting at 7.0 C with a 0.1 C deadband generates nothing until it drifts past 7.1 C. This alone strips out the noise floor of a stable process.

Swinging-door compression:

The archive stage then applies a linear algorithm, classically swinging door, that keeps a value only if a straight line drawn between the last stored point and the next candidate would deviate from the real signal by more than a compression deviation setting. On a ramp or a flat line it stores two endpoints and throws away everything between them; on a noisy or fast-changing signal it stores far more. The exact behaviour, and how aggressively it prunes, is vendor-dependent and tuning-dependent, so the compression ratios below are ranges I have seen in practice, not guarantees.

Typical compression ratios I plan against, hedged deliberately:

• Slow, stable analogue (tank levels, ambient temps): 20:1 to 100:1
• Active control loop PVs: 3:1 to 10:1
• Status / boolean tags: already tiny; change-of-state is the compression
• Energy meters at 15 min: little further gain, they are already sparse

A blended ratio of somewhere between 5:1 and 15:1 across a mixed tag population is a reasonable planning assumption. Confirm it against a pilot on your own signals before you commit budget. The OPC Foundation standards govern how most collectors pull data off the control system in the first place, and the collector's own exception settings are the first compression stage, so tune them together, not in isolation.

Step 4: The storage math, worked for 5k, 20k and 100k tags

Now put it together. The table below models three site sizes using a blended profile: a mix of the four tag classes above, a planning figure of about 8 bytes per stored event, and a conservative 10:1 blended compression ratio. Raw is what you would store with no compression at all; compressed is the realistic on-disk figure. Both are annual.

Read those as order-of-magnitude figures, not quotes. They assume a blended profile; a site that is genuinely mostly one-second control tags will land higher, a site dominated by meters and status bits will land far lower. The value of the table is the shape it reveals: raw storage is frightening, compressed storage is manageable, and the gap between the 5:1 and 10:1 columns is entirely a tuning decision you control. I always model both ends and budget toward the pessimistic one.

Step 5: Design the retention tiering

You do not keep everything at full resolution forever. You tier it. The design I default to has three layers.

Full resolution online: 13 months.

Thirteen months, not twelve, and the extra month is deliberate. It lets you compare this month against the same month last year with a full month of overlap, which matters for anything seasonal: chiller loads, heating, occupancy-driven energy. Keep the compressed full-resolution archive on fast, online storage for these 13 months so year-on-year analysis is a query, not a restore job.

Aggregated beyond 13 months.

Past the 13-month window, most reporting needs trends, not every sample. Roll the data up to hourly aggregates, typically min, max, average and a count or total, and keep those for several years. Hourly aggregates are a tiny fraction of the full-resolution volume, so multi-year trend retention costs almost nothing on disk. This is where long-term energy and performance reporting lives.

Archive policy.

Full-resolution archives that age out of the online window get moved to cheaper, slower, offline or near-line storage before deletion, with a retention period set by policy and by any regulatory or contractual requirement. Write the policy down: what is kept, at what resolution, for how long, and who signs off before anything is purged. An undocumented "we think it rolls off after a year" is how evidence disappears.

What aggressive compression quietly destroys

Here is the part the datasheets do not stress, and it is the reason I write this section in every historian design. Compression tuned for storage efficiency is optimised to reconstruct the shape of a signal within a deviation band. It is not optimised to preserve rare, short-lived events, and those rare events are exactly what reliability analysis needs.

What gets thrown away when you compress hard:

• Short transients that fall inside the deviation band but reveal a fault developing
• Start/stop events where the interesting behaviour is a few seconds of inrush or ramp
• Alarm spikes that flash and clear faster than the compression cares to store
• Anything whose diagnostic value is in the edge, not the steady state

This is the trap that catches people two years later. A pump fails, someone opens a reliability investigation, they pull the history for the weeks before the failure, and the data is smooth. Not because nothing happened, but because the compression that saved you 90 percent of your disk also erased the short transients and spikes that would have shown the fault coming. The historian did exactly what it was told. It was told the wrong thing for that tag.

This is also where the historian and the maintenance world meet. If a tag is going to feed condition monitoring or failure analysis, its retention and compression settings are a reliability decision, not just an IT one. I cover how that data crosses into the asset systems in SCADA to EAM integration, and the collection and buffering side in SCADA historian integration.

Licensing and the wider analytics warehouse

Storage is rarely the expensive part. Licensing usually is, and most enterprise historians license primarily per tag, sometimes in bands or blocks. That single fact changes the whole sizing exercise: every tag you historize "just in case" is a recurring licence cost, not a one-off gigabyte. Common models you will meet:

• Per-tag / point count, often sold in tiers, with a step-change in price at each band boundary
• Named collector or interface licences on top of the tag count
• Client / user or connector licences for analytics and visualisation access
• Enterprise or unlimited-tag agreements for very large estates, where the maths flips

Exact models are vendor-specific and change over time; the AVEVA PI System is the reference point most people know, but confirm current terms with the vendor for your own count. The takeaway is directional: because you pay per tag, tag discipline is a licensing lever, not just a storage one.

Finally, the historian does not live alone. Increasingly it feeds a wider analytics warehouse or data lake where operational data meets business data. That raises two questions worth settling early. First, what resolution does the warehouse actually consume? It is almost always aggregates, hourly or daily, not one-second raw, so you can send it the tiered data rather than replicating the full firehose. Second, where does compression happen in that pipeline? If the warehouse is doing your long-term reliability analytics, it inherits whatever the historian's compression already destroyed. The compression decision made for a single tag on day one echoes all the way out to the analytics platform years later.

A note on independence

I sell no historian and take no vendor commission. I have implemented and integrated these systems across process and building environments for over two decades, and the figures here are planning ranges from that experience, not vendor quotes. Swinging-door behaviour, per-tag storage sizes and licensing terms are all vendor-dependent and change between products and releases. Treat every number in this article as a starting range to validate with a pilot on your own signals and with your vendor's current documentation, not as a specification.

Conclusion

Historian sizing becomes a costable exercise the moment you stop treating it as one big number. Count the tags, split them into classes, choose a scan rate that matches how each class behaves, apply deadband and swinging-door compression with honest ranges, and the storage figure falls out. Then tier retention, 13 months full resolution online, hourly aggregates beyond, a written archive policy, and protect your reliability-critical tags from the compression that would otherwise erase the evidence you will one day need. Do the arithmetic before you buy the disk, and never let storage efficiency quietly overrule a reliability requirement.