The execution gap is not a mystery. It is the visible result of a single root cause: the work being run no longer resembles the work the infrastructure was built to run. Infrastructure was designed for predictable jobs you could place on a machine and leave there. Modern AI work breaks that assumption in three distinct ways, and the execution model never adapted to any of them.
These are not three problems. They are one shape change seen along three axes. Understanding them separately is what makes the gap legible, and what makes the fix obvious.
Shift one: the work became heterogeneous
The old unit of work was uniform. A web request, an API call, a database query, all of them ran on one kind of resource for their whole life. You sized a machine for the job and the job stayed inside it.
A modern AI workload is not one kind of work. It moves between CPU-heavy and GPU-heavy stages within a single job. A typical pipeline runs CPU-heavy preprocessing, then GPU-heavy inference or training, then CPU-heavy postprocessing.² The resource each stage needs is different, and the difference is large.
The execution model has no answer for this except to package the whole thing together and run it on one node. The consequence is mechanical and unavoidable. When CPU-heavy and GPU-heavy stages are packaged as a single workload, the entire workload is forced to scale as one unit, which means the expensive GPU stays allocated even while only the CPU stages are running. Anyscale's own illustration is exact: a container that does CPU preprocessing then GPU inference, deployed on a node with 8 GPUs and 64 CPUs, can saturate all 64 CPUs while the GPU stage uses only 20 percent of the GPUs. To get 5x more throughput you replicate the entire instance four more times, even though the GPUs were never the bottleneck, and you end up with 5 GPU nodes each using a fifth of their capacity.¹
This is the first source of the gap. The work has two shapes and the infrastructure offers one box. You pay for the most expensive resource across the whole lifecycle even though it is only needed for part of it.
The trend is sharpening, not softening. Agentic and reasoning workloads make the heterogeneity worse, because every tool call, environment step, reward evaluation, and orchestration decision runs on CPU while the GPU sits idle waiting for the next forward pass. Reinforcement learning is a multiplier on this: it introduces exploration, multiple rollouts, and environment orchestration, all CPU-and-I/O-bound work wrapped around the GPU work.³ The mismatch between the shape of the work and the shape of the box is widening exactly as the work becomes more valuable.
Shift two: the work stopped being a single step
The old unit of work was atomic. One request, one execution path, one resource, start to finish. You could reason about it as a single thing because it was a single thing.
Modern AI work is staged, and the stages have opposite resource profiles. The clearest case is the one the entire industry has now converged on. LLM inference is not one operation; it is two. Prefill is compute-bound. Decode is memory-bandwidth-bound. Run them on the same GPU and they fight each other, because the configuration that is optimal for one is wrong for the other. The result is idle compute during the mismatched phase, and it is structural, not a tuning error.⁴
The proof that this is a shape problem and not a software problem is who is now fixing it and how. NVIDIA shipped Dynamo to general availability at GTC 2026 specifically to split prefill and decode across dedicated hardware pools, reporting up to roughly 7x throughput gains on Blackwell from the separation. NVIDIA's own position is that co-locating the two phases on one GPU leads to inefficient resource use.⁵ Google Cloud publishes a disaggregated-inference recipe on the same principle.⁶ Microsoft, Anyscale, Gartner, and the FinOps Foundation describe the fix the same way: break execution into stages and let each run on the resource it actually needs.⁷
This is the second source of the gap. The old model takes the unit of work as given and places it. But when the unit is itself a sequence of stages with conflicting needs, placing the whole thing on one resource guarantees that some of that resource is wasted for some of the time. You cannot place your way out of a problem that lives inside the unit.
Shift three: the work stopped living in one place
The old unit of work had a home. It ran in one cluster, in one region, in one cloud. Locality was an assumption baked so deep it was never stated.
Modern AI work spans clouds, regions, clusters, networks, and edge environments, and increasingly it has no choice. GPU scarcity forced a "compute wherever available" posture: rather than wait months for concentrated allocation in one location, teams now assemble capacity across multiple regions, multiple public clouds, private data centers, and specialized providers.⁸ More than 80 percent of organizations now use multiple public cloud providers, and AI is sharpening the pressure further as sovereign-AI and data-locality requirements dictate where models and data are allowed to live.⁹ At the same time inference is moving back toward on-prem and edge for latency and data-locality reasons, so a single workflow can now straddle a training cluster in one cloud, an inference tier at the edge, and data sitting somewhere else again.¹¹
The execution model was built for one location and one resource type. It has no native concept of sending a unit of work across that fragmented landscape to the resource best able to run it. So teams move data to the compute or compute to the data, both slow, complex, and expensive, and the public internet they often fall back on was never designed for the predictable latency and east-west traffic distributed AI demands.¹⁰
This is the third source of the gap. The work is distributed by necessity and the execution model assumes it is local. Every workaround for that mismatch, replication, data movement, manual placement across providers, is a tax paid because the layer underneath cannot route work to the right place on its own.
One shape, three axes
Put the three together and the picture is coherent. The old workload was homogeneous, atomic, and local: one kind of resource, one step, one place. The modern workload is heterogeneous, staged, and distributed: many resources, many stages, many places. Every one of those changes happened inside the same few years, and the execution model answered none of them.
This is why low utilization, high cost, and rising complexity are not three separate failures. They are the same mismatch seen from three angles. The work changed shape on three axes at once, and the layer responsible for running it still assumes the old shape on all three.
It also explains why orchestration cannot close the gap. Orchestration decides where a unit of work is placed and when it moves. But every one of the three shifts is about the unit of work itself: what it is made of, how many stages it has, and where those stages need to run. A scheduler takes the unit as given. The shape change is a change in the unit. More scheduling on top of the same unit manages the symptom and leaves the cause untouched, which is exactly why measured utilization has gone backward in the environments where orchestration became the default.¹²
The fix follows from the diagnosis
If the problem is that the unit of work changed shape, the fix is to change the unit. Break a workload into smaller units, send each unit to the resource best suited to run it, and run it there. Match the CPU stages to CPU, the GPU stages to GPU. Send prefill and decode to the hardware each one needs. Route work across clouds, regions, and edge to the place that can actually run it, instead of assuming it stays home.
That is a layer beneath the scheduler and above the hardware. It does not compete with orchestration. The whole field has agreed on the destination. The shape change is the reason that destination exists, and reaching it fully means operating on the unit of work itself, on all three axes, rather than placing an unchanged unit better.
The gap is not a percentage that happens to be low. It is the precise distance between the shape of the work and the shape of the infrastructure. Close that distance and the gap closes with it.
A note on the numbers
The figures here are reported with their source and what they measure. Utilization claims distinguish fleet utilization, model-FLOPs utilization, and server-capacity utilization, because they are three different things. The Dynamo throughput figure is a vendor-reported benchmark on a specific model and platform and is labeled as such. The disaggregation evidence is drawn from the primary sources closest to the hardware, including NVIDIA, Google Cloud, and the analysts who reached the same conclusion independently. As with the execution gap, the argument does not rest on any single number. It rests on the shape of all of them together.
References
- Anyscale, GPU (In)efficiency in AI Workloads. Single-container packaging forces CPU and GPU stages to scale as one unit, guaranteeing low utilization; the 8-GPU / 64-CPU example with 20% GPU use and 5x replication. Anyscale
- GMI Cloud (Jan 2026). Multi-stage pipelines run sequentially, wasting capacity at each handoff. GMI Cloud
- Agentic and RL CPU pressure: tool calls, environment steps, and rollouts hammer CPU, with RL as a multiplier. Spheron CPU-to-GPU ratio guide (Jun 2026), fpx.world "Beyond the GPU" (Jan 2026), Viks Newsletter (Feb 2026). Spheron fpx.world Viks Newsletter
- Prefill compute-bound, decode memory-bandwidth-bound; idle compute on the mismatched phase. NVIDIA Dynamo design docs; Sarathi-Serve (arXiv 2403.02310). NVIDIA docs arXiv
- NVIDIA Dynamo GA at GTC on March 16, 2026; prefill/decode split with up to ~7x on Blackwell; co-location described by NVIDIA as inefficient resource use. NVIDIA Spheron Digital Applied
- Google Cloud disaggregated-inference recipe using NVIDIA Dynamo on AI Hypercomputer. Google Cloud
- Convergence on stage separation across Microsoft/Yotta Labs, Anyscale, Gartner, and the FinOps Foundation. Yotta Labs Anyscale FinOps Foundation Gartner
- "Compute wherever available": capacity assembled across regions, clouds, private data centers, and specialized providers. Alluxio geo-distributed GPU whitepaper. Alluxio
- More than 80% of organizations use multiple public clouds; sovereign-AI and data-locality pressures. AceCloud (Gartner-cited, Jan 2026); floLIVE (Multi-Cloud 2026). AceCloud floLIVE
- Distributed AI workloads are inherent, and the public internet was not designed for predictable latency or east-west traffic. Equinix (Jan 2026). Equinix
- Inference moving back toward on-prem and edge; single workflows spanning multiple locations. Nutanix (Mar 2026). Nutanix
- Utilization went backward where orchestration matured; ~5% average GPU. Cast AI 2026 State of Kubernetes Optimization Report; SDxCentral. Cast AI SDxCentral