The 5 percent is two problems
Average GPU utilization across tens of thousands of production clusters sits near 5 percent.¹² That figure hides two different problems, and they have to be separated before any recovery makes sense.
The first is fleet-level idle: capacity that is on and running nothing. Reserved against bursts that rarely arrive, held for projects that have gone quiet, allocated to teams that asked for more than they use. The second is in-workload waste: a GPU running a real job that still wastes most of the chip. These are different quantities, recovered at different layers, and they must never be added together. This piece is about the first. The second is its own walkthrough, Recovering capacity inside the workload.
Idle comes first because it is the biggest tranche and the cheapest to fix. Basic consolidation, packing real work onto fewer machines and reclaiming the idle ones, moves a 5 percent fleet toward a roughly 30 percent baseline.³ Most of the distance from 5 to a sane number is not an optimization problem. It is an accounting problem.
Why so much capacity sits idle
Idle at this scale is not negligence. It is the rational response to pressures that all push toward holding more than you use.
The biggest is defensive over-provisioning. When accelerators were rationed and lead times were long, the safe move was to grab capacity and hold it, because giving it back risked not getting it again. That instinct outlived the scarcity. Reserved-but-idle capacity is now a named, endemic cause of AI infrastructure waste, alongside static provisioning and holding whole accelerators against bursts that rarely arrive.³⁴
The second pressure is organizational. Capacity gets allocated to a team or a project, the project slows, and the allocation stays. Nobody is paid to hand back a GPU. The server-era version was the comatose machine: roughly 30 percent of physical servers did no useful work for six months or more, stranding billions in capital.⁵ The accelerator era repeats the failure with far more expensive hardware.
The third pressure is structural, and it connects idle to the rest of the thesis. The scheduler places a unit of work on a node but does not break it apart. So a job that is mostly CPU work, with a GPU stage in the middle, holds a GPU node for its whole life. The dashboard shows the accelerator assigned; it is idle most of the wall-clock time, because the work on it does not need it most of the time. Consolidation alone cannot reclaim that GPU, because the packaging holds it hostage.
What consolidation recovers, and where it stops
The first recovery is unglamorous: count what is on, find what is doing nothing, reclaim it. Bin-packing real work onto fewer machines and turning off what nothing uses moves a 5 percent fleet toward roughly 30 percent with no change to how any workload runs.³ This tranche needs no exotic technology, only the decision to stop paying for idle and the tooling to see which capacity is truly idle versus merely reserved.
Then it hits a ceiling, and the ceiling is the structural pressure above. You can reclaim a GPU allocated to nothing. You cannot, by packing alone, reclaim a GPU allocated to a job that needs it twenty percent of the time, because the job holds the whole node. Recovering that capacity means changing what is scheduled, not just where it lands.
Routing across the fleet recovers the rest
The capacity consolidation cannot reach comes back only by changing the unit of execution: breaking a workload into smaller units, sending each to the resource best suited to run it, and running it there, across heterogeneous hardware.
When the CPU stages of a job land on CPU capacity anywhere in the fleet instead of riding a GPU node, the GPU stages consolidate onto fewer accelerators that stay fed, and the reserved-but-idle tranche shrinks toward the work that is present. A burst no longer forces a whole-accelerator reservation that sits idle between bursts. Capacity in one location can absorb demand from another. The idle that was structural becomes reclaimable.
This is a layer beneath the scheduler and above the hardware. It does not replace the scheduler, which keeps deciding intent. It changes what the scheduler is scheduling, so placement no longer reserves the most expensive resource for the whole life of a job that needs it for part. The same change that lets a workload run on the right resource is what lets the fleet stop holding the wrong resource idle.
The honest size of the prize
The idle tranche is large, but overstating it is how these claims lose credibility. A fleet at 5 percent has room to several-fold its useful output before it touches any physical ceiling. The first chunk, the move toward roughly 30 percent, comes from consolidation that needs no new hardware and no workload rewrite.³ The rest comes back through routing and the unit-of-work change. Neither is a promise of a specific number for a specific fleet, because the mix of free-idle versus structurally-held capacity differs by shop. What holds everywhere is the shape: most of the gap between 5 percent and a sane baseline is capacity that is on and idle, and much of it needs only to be counted and reclaimed.
That is the cheapest capacity in the market. When the next megawatt is years out, the idle accelerator on the floor today is the only capacity that arrives on demand.
The point
The 5 percent number is idle capacity and in-workload waste stacked together, and the idle half is larger and cheaper to recover. Some of it is plainly doing nothing and comes back through consolidation. The rest is held hostage by packaging that pins expensive resources to jobs that need them part of the time, and it comes back only by changing the unit of execution so work routes to the resource that fits it. None of this requires buying a single accelerator. The capacity is already there, powered and paid for, waiting to be turned into work.
A note on the numbers
The figures here are reported with their source and what they measure. Fleet utilization, model-FLOPs utilization, and server-capacity utilization are three different things; every number in this piece is fleet-level, and the in-workload figures from the sibling walkthrough are kept out so the two are never added. The roughly 30 percent consolidation baseline is a no-effort estimate against a measured ~5 percent average, not a guaranteed outcome, because the split between free-idle and structurally-held capacity varies by shop. The comatose-server figure is from the server era, cited as a recurring pattern, not a direct GPU measurement. As across the series, the argument does not rest on any single number. It rests on the shape of all of them together: most of the gap below a sane baseline is capacity that is on and doing nothing.
References
- Cast AI, 2026 State of Kubernetes Optimization Report. Average GPU utilization of 5 percent across tens of thousands of production clusters (AWS, Azure, GCP), with both CPU and GPU down year over year. Cast AI
- Independent reporting on the Cast AI findings: roughly 5 percent average across about 23,000 clusters, roughly 20x over-allocation. ITBrief. ITBrief
- Defensive over-provisioning under scarcity drives reserved-but-idle capacity; a no-effort consolidation baseline near 30 percent versus measured ~5 percent. Rack2Cloud analysis of the Cast AI data. Rack2Cloud
- FinOps Foundation, FinOps for AI Working Group. Names GPU underutilization, overprovisioning, and static provisioning as endemic causes of AI infrastructure waste; recommends reserving a true baseline and bursting on demand rather than holding idle accelerators. FinOps Foundation
- Koomey / Anthesis Group. Roughly 30 percent of physical servers found "comatose," doing no useful work for six or more months, stranding billions in capital; enterprise utilization rarely exceeding single digits. Koomey Anthesis