The reflex
Average GPU utilization near 5 percent is no secret. The hyperscalers measure it. NVIDIA measures it. Every infrastructure team measures some version of it.⁷ So the reflex is obvious: if a layer could recover most of a multi-hundred-billion-dollar asset base, a company with NVIDIA's hardware or a hyperscaler's fleet would have shipped it. The number is still 5 percent, so the problem must be harder than it looks.
The problem is not harder than it looks. It sits where the incumbents are built not to reach. Three reasons.
Reason one: the control plane was inherited, not designed
AI never got a control plane built for accelerators. It inherited Kubernetes, built for stateless CPU services, and bolted GPUs on.¹
The original model treated a GPU as an opaque integer: one per node, no notion of memory, topology, or how busy it was.¹² A pod asked for a whole GPU and got one, even if it used the card a fifth of the time.³ Everything since is a patch on that inheritance. Dynamic Resource Allocation went GA in Kubernetes 1.34 and lets a workload request a GPU by memory and topology.⁴⁵ KAI added gang scheduling. Real improvements, and still the same job: decide where to place a pod, not change what the pod is.⁶ You can perfect placement for a decade and never touch what the model holds fixed.
Reason two: the waste lives below the layer everyone improved
Every one of those improvements works on the pod, picking where the bundled unit lands. The waste is created inside the unit, where CPU stages and GPU stages are packaged together and the expensive accelerator is held for the whole life of a job that needs it for part. Placement cannot reach in there. That is not a tuning gap. It is the definition of placement. You cannot place your way out of a problem that lives inside the unit.
The proof is the trend. If better scheduling were going to close the gap, utilization would be climbing as the tooling matured. It is falling. Across tens of thousands of clusters, efficiency is going backward as Kubernetes adoption grows, and the gap between what teams pay for and what they use is widening.⁷⁸ The most capable teams in the industry have thrown more orchestration at the same unit, at massive scale, and the number got worse.
The fixes that work ship outside the scheduler. NVIDIA's own answer to inference waste, Dynamo, does not place better. It splits prefill from decode and runs each on different hardware, a change to the unit, shipped as a separate framework.⁹ The field's best move against the gap already lives below the orchestrator, because that is where the gap is.
Reason three: the people best placed to fix it get paid either way
State this flatly, no motive-reading. The companies with the most hardware and the clearest view of the waste have the least reason to close it, because of how the asset is sold.
A buyer on a hyperscaler pays for allocated capacity, not work delivered. On-demand and reserved, billed by the GPU-hour the instance is held, at three to six times specialist rates.¹⁰¹¹ The bill is identical at 5 percent or 95 percent. The waste sits on the buyer's side of the meter, and the per-second billing that would expose idle time is a neocloud and spot feature, not the enterprise default.¹¹ NVIDIA's incentive is the next chip, with trillions in forward demand and supply already booked.¹² A layer that squeezes more work out of hardware already bought is not what a chip vendor or a capacity reseller leads with. No bad faith required. The party that sees the waste is not the party that pays for it. The buyer is.
So the fix comes from below
Stack the three. Inherited architecture, improved within its own limits. Waste below the layer those improvements touch. And the players who could build the fix are paid whether it ships or not.
A layer beneath the orchestrator and above the silicon clears all three. It does not inherit the pod as fixed, because changing the unit is the job. It does not fight the scheduler, so it needs no incumbent to rebuild a control plane. And it is measured where the waste lands, on hardware the buyer already paid for, which puts it on the same side as the party carrying the cost. This is not unsolved because it is impossible. It is unsolved because solving it means standing where the incumbents are not built to stand.
The point
The gap is not open because it is invisible or because the recovery is small. It is open because the control plane was inherited, the waste lives below the layer everyone keeps improving, and the firms best placed to see it are paid either way. Three structural facts, not a lack of effort. They also point at the fix: a neutral execution layer, below the scheduler, that changes the unit and is measured on the buyer's side of the meter. You cannot place your way out of a problem that lives inside the unit, and the people who sell placement were never going to be the ones to prove it.
A note on the numbers
Figures are reported with their source and what they measure. The 5 percent figure is fleet-level average GPU utilization across tens of thousands of production clusters, and the claim that it is widening is from the same report, a measured trend, not a projection.⁷ The pricing comparison, hyperscalers at three to six times specialist rates, is a Q2 2026 market observation and will move. NVIDIA's forward demand and supply commitments are from its own filings.¹² The argument is structural, not quantitative: it rests on the direction of the trend and the placement of the meter, not the precise size of any number. As across the series, the argument does not rest on any single number. It rests on the shape of all of them together.
References
- AI infrastructure inherited the Kubernetes control plane before Kubernetes understood accelerators; the integer device model, absence of topology awareness, queue-depth visibility, and model residency, and placement logic built for stateless CPU workloads, are the assumptions GPU work is now forced to operate within. Rack2Cloud (May 2026). Rack2Cloud
- GPU scheduling stuck at integer resource counts since the device-plugin model shipped in 2017, advertising GPUs as opaque integers per node with no notion of memory or topology. Spheron Kubernetes GPU orchestration guide (Apr 2026). Spheron
- The scheduler has no concept of actual utilization: a pod requesting four GPUs gets four even if it uses them 20 percent of the time. CNCF / HPE engineers on reclaiming idle GPUs (Jan 2026). CNCF
- Dynamic Resource Allocation graduated to general availability in Kubernetes 1.34, replacing the rigid device-plugin model with declarative hardware requests for GPU type, memory capacity, and interconnect topology. CIO (Apr 2026). CIO
- DRA combined with KAI Scheduler reflects a shift toward attribute-rich, scheduler-visible resources and topology-aware placement, donated to the CNCF at KubeCon EU 2026. Spheron; Rafay on the KubeCon EU 2026 contributions. Spheron Rafay
- DRA is a meaningful improvement but primarily helps Kubernetes decide where to place a pod; KAI and similar GPU-aware schedulers are valuable building blocks that still rely on static policies and manual configuration. ScaleOps (Dec 2025). ScaleOps
- The efficiency gains Kubernetes was designed to unlock are not emerging with scale; the gap between what organizations pay for and what they use is widening as adoption accelerates, with average GPU utilization at 5 percent. Cast AI 2026 State of Kubernetes Optimization Report. Cast AI
- Measured year-over-year decline in utilization in the environments where orchestration became the default. SDxCentral on Kubernetes efficiency going backward. SDxCentral
- NVIDIA Dynamo splits prefill and decode across separate hardware pools, a change to the unit of execution shipped as a standalone inference framework rather than a scheduler feature; reached general availability at GTC on March 16, 2026. NVIDIA Dynamo product page and design docs. NVIDIA NVIDIA docs
- Standard hyperscaler GPU billing is by allocated capacity (on-demand and reserved tiers, billed by GPU-hour held), independent of how much useful work the device delivers. AIMultiple cloud GPU provider comparison (Q2 2026). AIMultiple
- Hyperscalers priced three to six times above specialist neoclouds for the same GPU because rates bundle enterprise SLA and commitments; per-second usage-based billing that would expose idle time is mainly a neocloud and spot-tier feature. AIMultiple; RunPod cloud GPU provider guide. AIMultiple RunPod
- NVIDIA forward demand measured in the trillions with multi-year supply and cloud-service commitments already booked (purchase commitments of $119B and multi-year cloud commitments of $30B as of April 2026). NVIDIA FY2026 Q1 filings. NVIDIA 10-Q