Planning NGS data storage and the infrastructure around it is the part of running a sequencing program that most often gets deferred until it becomes a problem, and it is the one part that behaves unlike everything else in the budget. Every other cost in a sequencing lab, reagents, staff time, instrument service, scales with how much sequencing you do this year. Data storage does not. Each year of retained data is added to the last rather than replacing it, so a lab running a steady volume with full retention watches its storage cost climb year after year while its sequencing output stays flat. The data does not leave, and the bill compounds.
This overview covers the four decisions that keep data infrastructure from becoming the limit on capacity: how to size storage against run volume, how to tier and locate it economically, how to provision compute and select analysis tools, and how to set a retention policy and staff the analysis that turns data into results. The precise storage-accumulation figures, in terabytes per year and cumulative over five years, are modeled in Next-Generation Sequencing in the Lab: A Manager’s Guide to Building, Budgeting, and Scaling NGS Capacity; this article is about managing the problem those numbers describe.
Key Takeaways
|
Sizing Storage Against Run Volume
Storage planning starts from a number you can calculate: the data each sample generates, multiplied by your run volume, accumulated across the years you retain it. A single whole-genome sample generates a substantial set of files across its raw output, aligned data, and variant calls, and that per-sample figure multiplied by annual volume gives the yearly increment. The per-sample data volumes for different applications and depths can be sized with the Sequencing Coverage and Cost Calculator: How Much Data Do You Actually Need?, and the full accumulation model, showing how the yearly increment compounds into a much larger multi-year total under full retention, is in the main guide referenced above.
Two moves change the shape of that curve. Compression is the first: a study of CRAM compression across sequencing platforms found compression of roughly 40% to 70% for aligned data depending on platform, without altering variant calls, which can nearly halve the storage footprint of the largest data class. The second is retention policy, covered below, which determines how many years of accumulated data you are actually paying to keep. Sizing storage without deciding retention first is sizing for infinite accumulation, which no budget survives.
Tiering: Hot, Warm, and Archive
Not all sequencing data needs to sit on fast, expensive storage, and the biggest single lever on storage cost is matching each class of data to the right tier by how often it is actually accessed. Storage tiers trade access speed for price: hot storage is fast and expensive, archive storage is slow and cheap, and warm storage sits between. The instinct is to move everything possible to the cheapest tier, and that instinct is a trap, because the cheapest tier by storage rate is frequently the most expensive by total cost.
Tier | What Lives Here | Access Pattern | The Catch |
Hot | Data in active analysis; recent results | Frequent, immediate | Highest storage rate, but no access penalty |
Warm | Recently completed projects; occasional reference | Occasional | Moderate rate; small retrieval and minimum-duration terms |
Archive | Data you expect never to touch again | Rare to never | Cheapest to store, expensive and slow to retrieve |
Table 1. Storage tiers matched to genomic data by access pattern. The archive tier’s low storage rate is offset by retrieval fees and minimum-duration charges, so it only wins for data that genuinely stays untouched.
The Cheapest Tier Is Not the Cheapest Total CostDeep-archive cloud storage can cost around a dollar per terabyte per month, against roughly twenty-plus dollars per terabyte for hot storage, a difference that makes archiving everything look obvious. The catch is on the way back out. Archive tiers charge retrieval fees, on the order of tens of dollars per terabyte, and impose minimum storage durations of several months, so an object deleted or retrieved early still bills for the full minimum. For data you genuinely never touch, archive wins decisively. For data accessed even occasionally, the retrieval fees and minimum-duration charges can make a cheap tier cost more in total than a more expensive one would have. Match the tier to the real access pattern, not to the headline storage rate, and the savings are real; ignore retrieval cost and the cheap tier quietly becomes the expensive one. |
Cloud, On-Premise, and Hybrid
Where the storage physically lives is a separate decision from how it is tiered, and it turns on cost structure, control, data transfer, and the shape of your access needs rather than on a simple cheaper-or-not comparison. On-premise storage is a capital purchase you own and control, with predictable cost and no per-access charges, but it requires space, power, cooling, maintenance, and a refresh cycle, and it scales in chunks rather than smoothly. Cloud storage is an operating cost that scales smoothly and offloads the maintenance, but it introduces per-access and data-transfer charges, and moving large genomic datasets in and out has both a cost and a time dimension that catch labs off guard.
Interested in lab tools and techniques?
Many sequencing labs land on a hybrid: active data on fast local storage close to the compute that analyzes it, with older data tiered out to cheaper cloud archive under a lifecycle policy. This keeps the frequently accessed data fast and free of per-access charges while pushing the long tail of rarely touched data to the cheapest durable storage. The detailed cost-and-control comparison, including the data-transfer economics that often decide it, is developed in Cloud vs. On-Premise for Genomic Data: A Cost and Control Comparison, and the specific work of sizing the footprint for your volume is in How Big Is NGS Data? Sizing Storage for Your Sequencing Volume.
Choosing Analysis Software
Storage holds the data; analysis software turns it into results, and the two have different demands that are easy to conflate. Compute for sequencing analysis behaves differently from storage in a way that shapes how you provision it: storage demand is steady and cumulative, while compute demand is spiky, concentrated in the hours after a run completes when primary and secondary analysis run. That burst pattern suits elastic, on-demand capacity well and suits a fixed on-premise cluster sized for average load poorly, since the cluster sits idle between runs and then cannot keep up at the peak. The connection between run scheduling and this compute burst is part of the throughput picture in Running NGS at Scale: Throughput, Scheduling, and Automation.
Selecting the analysis software itself is a decision about fit, support, and validation rather than raw capability. The questions that matter for a lab manager are whether the software supports the applications you run, whether it fits the skills of the people who will operate it, how it is supported and updated, and, for regulated work, whether its versions and outputs can be controlled and documented. Choosing analysis tools is a substantial decision in its own right, developed in Choosing Genomic Data Analysis Software.
Retention, Backup, and Deletion Policy
Retention policy is where the compounding storage problem is actually controlled, and it is a budget decision that too often gets made by default rather than by design. The core questions are what you keep, in what format, for how long, and on what tier. A workable default for many research labs keeps processed results and variant calls indefinitely on accessible storage, keeps aligned data in a compressed format on a lower tier, and retains raw output only for a defined window unless a specific requirement says otherwise. That approach preserves what is scientifically most valuable and most expensive to regenerate while letting the largest, most reproducible data class age out.
The constraints on that default are external and have to be checked before the policy is written, not after. Funder and journal data-availability requirements, institutional records policy, and sample consent terms all bear on what must be kept and for how long, and some of them require retention or deposit that overrides a lab’s cost-driven preference. Backup is a separate requirement from retention: retention decides what you keep, backup ensures a second copy survives a failure, and the two are often confused to the detriment of both. The full policy, including backup strategy and defensible deletion, is developed in Data Retention, Backup, and Archiving Policy for Sequencing Labs. Deciding this before the first run, rather than after the first invoice that makes someone ask, is the single most effective control on the storage line.
Who Analyzes the Data
The most important resource in a sequencing program’s data infrastructure is not storage or compute; it is the people who turn completed runs into answers, and it is the resource most consistently underestimated. Programs that scale sequencing capacity without scaling analysis capacity do not stop producing data. They accumulate unanalyzed runs, which is worse than a backlog of samples, because the money has already been spent and the result has still not been delivered. Storage fills up visibly and prompts action; an analysis backlog is quieter and often grows unnoticed until someone asks why completed runs have not produced results.
The honest staffing question is not how many people it takes to run the instrument. It is who converts a completed run into an interpretable result, how long that takes, and what happens to the queue when that person is unavailable. A program that depends on a single analyst has a single point of failure that stops result delivery entirely when that person is on leave. Planning analysis staffing deliberately, as a distinct function rather than an assumed side capacity, is covered in Who Analyzes the Data? Staffing Bioinformatics for a Sequencing Lab. It is the part of data infrastructure most likely to become the real constraint, and the part a storage-focused plan is most likely to miss. How the cost of all of this fits into the full program budget is in How Much Does NGS Cost? Budgeting Instruments, Reagents, and Sequencing.
This article was produced under Lab Manager's AI Editorial Guidelines.
