Recovering Capacity Inside the Workload

The waste that survives a busy GPU

A fleet at 5 percent has two problems inside that number. One is capacity that is on and running nothing, recovered by counting and consolidation, covered separately in Reclaiming idle capacity. The other is here: a GPU running a real job that still wastes most of its own capacity, because of how the job is packaged and served.

This is the harder tranche, and where changing the unit of work pays off. Even a well-tuned training run reaches only 35 to 45 percent of theoretical peak, and unoptimized inference commonly runs 20 to 40 percent.¹² The busy GPU is a stack of losses, and they come off in order, because each one changes the denominator for the next.

The representative pipeline

Take a workload that looks like most real ones. A request comes in, gets CPU-heavy preprocessing, runs an LLM inference step on a GPU, then gets CPU-heavy postprocessing. It is deployed the ordinary way: the whole thing in one container, replicated across GPU nodes, sized for peak, fronted by a scheduler that decides which node each request lands on. Measured with care, the GPU is busy a low double-digit fraction of the time at best. Four recoverable losses are stacked inside it. Take them in the order you would attack them.

Stop scaling CPU and GPU as one unit

The single container is the first and largest loss. Pre- and postprocessing are CPU work. Inference is GPU work. Packaged together, they scale together, so the expensive GPU stays allocated through the CPU stages even though it is idle for all of them.

Anyscale's example is exact: a container doing CPU preprocessing then GPU inference, on a node with 8 GPUs and 64 CPUs, can saturate all 64 CPUs while using only 20 percent of the GPUs, and getting 5x more throughput means replicating the whole instance four more times, ending with five GPU nodes each at a fifth of capacity.³ Split the stages so CPU work runs on CPU and GPU work runs on GPU, and the GPU stops waiting on preprocessing it should never have held. This is the largest single recovery in most pipelines, because it attacks the structural mismatch directly, not at the margin.

Batch the GPU work continuously

Once the GPU stage runs on its own, the next loss is in the serving engine. Static batching waits for a whole batch to finish before admitting new requests, so slots sit idle until the slowest one completes. Measured padding overhead runs 60 to 80 percent for typical workloads, and you can watch it: a static-batched server shows SM utilization around 30 to 40 percent even at moderate request rates.⁴

Continuous batching works at the iteration level. When a request finishes, its slot frees immediately and the next one takes it. Reported result: utilization climbing from 30 to 40 percent to 75 to 85 percent on a mixed workload, with 4 to 8x throughput when output lengths vary.⁴⁵ In modern engines this is on by default, so fleets leaving it unclaimed are usually just misconfigured.⁴

Stop wasting the memory the work runs in

Inference is gated as much by memory as by compute. Older serving systems reserved contiguous key-value cache regions in advance, fragmenting memory until effective utilization fell to around 20 percent.⁶ PagedAttention stores the cache in fixed-size blocks mapped to non-contiguous memory, dropping waste to under 4 percent and letting far more requests share a card.⁶⁷ More requests in flight means more to batch, feeding back into the gain above.

A second lever: quantizing weights from 16-bit to 8-bit roughly halves weight memory, and 4-bit cuts it further, freeing room for a larger cache and higher concurrency at a quality cost that is small for many workloads.⁵⁸ On shared-prefix workloads, agentic loops that repeat a system prompt, RAG calls that reuse context, prefix caching reuses the cached computation instead of recomputing it, reported at up to 5x faster inference and 30 to 60 percent lower time-to-first-token.⁷⁹

Separate the two phases of inference

The inference step is not one operation. Prefill processes the prompt and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. On the same GPU they interfere, because the configuration that suits one is wrong for the other, and one phase starves while the other saturates.⁵¹⁰

This is the loss the whole field converged on. NVIDIA shipped Dynamo to general availability in March 2026 to separate prefill and decode across dedicated hardware pools, reporting up to roughly 7x throughput on Blackwell, and Google Cloud publishes a recipe on the same principle.¹⁰¹¹ The gain is large, but it pays most once the cruder losses above it are closed, which is why it comes late. A pipeline that separates phases while still scaling CPU and GPU as one unit has fixed the subtle problem and left the obvious one in place.

Adding it up

The losses do not multiply cleanly, and anyone who says they do is selling. Each move changes the denominator for the next, gains overlap, and the ceiling is set by the workload.

Inside an active inference workload, continuous batching plus paged memory takes a 30 to 40 percent server into the 75 to 85 percent range.⁴ Phase separation and the memory levers add throughput on top, workload-dependent, and the most aggressive demonstrations show a single well-tuned model saturating an accelerator near 100 percent under ideal conditions, a ceiling to cite carefully, not a number to promise.⁵ Training has its own ceiling: 35 to 45 percent MFU is a good run, and the gap there is mostly communication and pipeline overhead, not idle time you can reclaim.¹

The realistic target is not 100 percent. It is a defensible multiple of where workloads start: a server at 30 to 40 percent has clear room to roughly double its useful output through batching and memory work alone, before phase separation adds more. The ceiling matters less than the distance to it, and the distance is wide.

Where this connects to the fleet

Everything above recovers a single workload on a single node or pool. It does not touch the larger tranche, the capacity that is on and running nothing across the fleet, which comes back through consolidation and routing, not serving-engine tuning. The two halves meet at the unit of work: splitting CPU from GPU inside a pipeline is the same move, up close, that lets the fleet stop pinning expensive accelerators to jobs that need them part of the time. Recovering capacity inside the workload and reclaiming idle capacity across the fleet are one change at two scales: send each unit of work to the resource that fits it, and run it there.

The point

Recovery inside a workload is mechanical, not magical. The busy GPU wastes most of its capacity through a stack of losses that come off in order: split CPU from GPU, batch continuously, fix the memory, separate the inference phases. Each layer has a measured gain and a real ceiling, the gains do not stack into a headline number, and the honest target is a defensible multiple of where workloads start, not a saturated chip. None of it requires buying a single accelerator. It requires running the work you already have on the resource it needs.

A note on the numbers

The figures here are reported with their source and what they measure, and they are all in-workload, server-level numbers, kept separate from the fleet-level idle figures in the sibling piece, because adding the two as if they were the same is the most common way these walkthroughs mislead. Fleet utilization, model-FLOPs utilization, and server-capacity utilization are three different things; the 75 to 85 percent figure is server-level inside an active workload, and the 35 to 45 percent figure is MFU on training. The continuous-batching, PagedAttention, prefix-caching, quantization, and Dynamo figures are vendor and practitioner benchmarks on specific models and hardware; they do not stack multiplicatively, and the walkthrough does not multiply them. The near-100 percent saturation figure is a best-case demonstration, cited as a ceiling, not a target. As across the series, the argument does not rest on any single number. It rests on the shape of all of them together: a stack of in-workload losses that each come off at a known layer.

References

1. 35 to 45 percent MFU on well-tuned training runs, with the remaining gap dominated by communication and pipeline overhead. Meta Llama 3 (arXiv 2407.21783); CoreWeave H100 benchmarks. arXiv CoreWeave
2. Production LLM inference commonly observed at 20 to 40 percent GPU utilization before optimization. Yotta Labs. Yotta Labs
3. Anyscale, GPU (In)efficiency in AI Workloads. The 8-GPU / 64-CPU single-container example: 64 CPUs saturated while GPUs run at 20 percent, and 5x throughput requiring full-instance replication to five nodes each at a fifth of capacity. Anyscale
4. Static batching padding overhead of 60 to 80 percent and SM utilization of 30 to 40 percent; continuous batching, on by default in modern engines, lifting utilization to 75 to 85 percent. Spheron LLM serving optimization (Apr 2026). Spheron
5. Microsoft, The LLM Inference Optimization Stack. Continuous batching pushing utilization from 30 to 40 percent toward 80 percent; quantization roughly halving weight memory from 16-bit to 8-bit; the optimization stack including disaggregated inference and MIG partitioning. Continuous-batching analysis documenting 4 to 8x on mixed workloads and near-100 percent accelerator saturation as a best-case demonstration. Microsoft TianPan
6. PagedAttention (Kwon et al., 2023). Older contiguous-reservation serving fell to around 20 percent effective memory utilization; block-based paging drops waste to under 4 percent. Reproduced via the vLLM and PagedAttention literature. PagedAttention
7. RadixAttention and prefix reuse delivering up to 5x faster inference on shared-prefix workloads; block-based paging enabling far higher concurrency per card. TianPan continuous-batching analysis (May 2026). TianPan
8. Quantization (INT8/INT4) cutting model memory 2 to 4x with limited quality loss for many workloads, freeing room for larger KV cache and higher concurrency. buildmvpfast batching analysis (Mar 2026). buildmvpfast
9. Prefix sharing in agentic and RAG pipelines reducing time-to-first-token by 30 to 60 percent versus naive per-request serving. RunPod LLM inference optimization playbook (Mar 2026). RunPod
10. Prefill compute-bound, decode memory-bandwidth-bound; co-location causes interference. NVIDIA Dynamo reached general availability at GTC on March 16, 2026, shipping prefill/decode separation with up to roughly 7x throughput on Blackwell. NVIDIA Dynamo design docs and product page. NVIDIA docs NVIDIA
11. Google Cloud disaggregated-inference recipe using NVIDIA Dynamo on AI Hypercomputer. Google Cloud