A multimillion-dollar GPU investment can be underperforming before the first model finishes training. The problem may not be a shortage of compute but It may be that the storage and data infrastructure feeding that compute cannot deliver data quickly or consistently enough to keep the GPUs working.
Meta’s experience at hyperscale illustrates the problem, In July 2026 engineering report, the company said storage bottlenecks remain one of the primary contributors to GPU stalls in AI workloads. Its training environment involves hundreds of thousands of GPUs repeatedly reading large datasets, and when one GPU experiences a high-latency storage fetch, synchronization can slow the entire training step. Meta redesigned parts of its storage architecture and reported an average 80% cache hit rate from a distributed data cache, reducing the amount of data that had to reach storage directly.
The lesson is bigger than Meta, As enterprises spend heavily on accelerated computing, GPU utilization is becoming partly a storage architecture question.
For CIOs, the more important question before adding another tranche of GPUs is therefore: Are we actually compute-constrained, or are our GPUs waiting for data?
When Expensive Compute Is Waiting on Storage
AI training puts unusual pressure on storage. GPUs consume data in parallel, repeatedly reading training datasets while the system also handles model weights, intermediate files and checkpoints. The storage layer has to absorb these requests without allowing latency spikes to interrupt computation.
So, Meta found that its older storage architecture, which had evolved for conventional workloads, introduced metadata lookups and proxy layers that could push latency into the hundreds of milliseconds. That might be acceptable for many traditional applications, but it becomes a serious problem when an AI workload expects predictable, low-latency access to flash storage. Meta subsequently consolidated its metadata architecture and removed a data-plane proxy so storage servers could stream data directly to clients.
This distinction matters because average throughput alone can hide the problem. A storage platform may report impressive bandwidth while individual requests still experience enough latency to stall parts of a distributed training job.
Moreover, checkpointing creates another pressure point, training periodically writes model state back to storage, producing bursts of I/O that can compete with the reads needed to keep the next training batches moving. At scale, these are not isolated storage events; they become part of the compute schedule.
The result is an uncomfortable infrastructure equation: the performance of the GPU cluster can be limited by the consistency of the data path feeding it.
GPU Utilization Has Become an Infrastructure Economics Metric
That makes GPU utilization more than an engineering statistic. If an accelerator spends part of its time waiting for data, the organization is paying for compute capacity it cannot fully use. And unlike a storage bottleneck, adding more GPUs does not necessarily solve the problem. It can make it worse by increasing concurrent demand on the same data path.
Stability AI provides a useful real-world example. The company found that limitations in its cloud-based Lustre environment were leaving GPU infrastructure idle and creating unexpected storage costs. After changing its storage architecture, Stability AI reported 93% GPU utilization, a 95% reduction in storage cost per terabyte and 35% faster model training. The company also reported that its training epochs were reduced by roughly three weeks on average. The important point is not the particular storage technology Stability AI selected. It is the economic relationship the example exposes: improving the data layer changed the productivity of an existing GPU investment.
That should influence how AI infrastructure is evaluated. Storage cannot be assessed only by capacity or cost per terabyte when the business case for the platform depends on accelerator utilization.
The Architecture Has to Follow the AI Workload
This is where AI storage architecture begins to look different from conventional enterprise storage.
Training workloads need high-throughput parallel access, predictable latency and efficient movement between datasets, caches and accelerators. Parallel file systems can support highly concurrent access; object storage provides durable scale for large datasets and data lakes; local NVMe and distributed caching can keep frequently accessed data closer to compute. The right combination depends on workload behaviour rather than a single preferred storage model.
Meta’s architecture is instructive because it uses several layers rather than asking one storage system to do everything. Its redesigned environment combines regional storage, on-host memory and flash caching, while prefetching brings data closer to GPUs before it is needed. Meta reported that its distributed cache absorbs traffic spikes and achieves an average 80% cache hit rate.
That approach also addresses a less obvious cost: data movement.
Meta noted that researchers can spend hours ingesting datasets into the region where GPUs are available. At that point, the storage architecture is affecting not only GPU utilization but research velocity, the number of experiments a team can complete in a given period.
For enterprise AI, the same principle applies. If every training run requires large-scale copying, preprocessing or movement before compute can begin, the organization is paying for infrastructure and waiting time that may never appear on the GPU invoice.
What Should a CIO Measure Before Buying More GPUs?
The answer is not to stop investing in GPUs. It is to make sure the infrastructure surrounding them can justify the investment.
That means looking beyond GPU utilization as a single dashboard number. Technology leaders should be able to see data-loading time, storage latency, I/O throughput, queue depth, checkpoint duration, cache effectiveness and the amount of data being moved between storage tiers or locations alongside accelerator utilization.
Those measurements help distinguish a genuine shortage of compute from a bottleneck somewhere else in the pipeline.
They also change the investment conversation. If GPUs are consistently waiting on storage, another GPU cluster may increase capital expenditure without increasing useful output. If the bottleneck is data delivery, improving storage throughput, locality, caching or the network between data and compute may produce more value from infrastructure that has already been purchased.
That is a different way of thinking about AI infrastructure: optimize the system before expanding the component.
The New AI Infrastructure Equation
The rapid growth of accelerated computing is making the old separation between compute and storage increasingly difficult to defend. AI workloads do not consume compute independently of data; they consume compute through a continuous data pipeline.
Meta’s experience demonstrates what happens when that pipeline becomes the constraint. The company did not simply add more hardware to compensate for GPU stalls. It changed metadata handling, eliminated unnecessary data-path overhead, introduced distributed caching and placed storage closer to geographically distributed GPU resources.
For Open Storage Solutions, this is the more important infrastructure conversation. AI storage has to be evaluated in terms of how effectively it turns expensive compute into productive work, throughput, latency, data locality, scalability and the ability to handle the access patterns created by training and inference.
The takeaway is before approving another major GPU investment, establish where the current environment is actually constrained. If the answer is storage, networking or data movement, adding compute may only make an expensive bottleneck more expensive.
The future of AI infrastructure will not be determined by who owns the most GPUs. It will be determined by who can keep those GPUs consistently productive.
Add your first comment to this post