Change the Unit of Work

One question separates two kinds of work

There is a single question that sorts the entire AI infrastructure stack into two piles. Does this layer decide where and when a unit of work runs, or does it change what the unit of work is?

Almost everything lives in the first pile. Orchestrators, batch schedulers, next-generation placement engines, serving frameworks, even the disaggregation systems the whole field is now converging on, all of them take a unit of work as given and get better and better at routing it, packing it, batching it, and timing it. That is placement. It is valuable, it is hard, and the engineering in it is real. But it shares a ceiling, because placement cannot recover waste that is created inside the unit before placement ever begins. This piece walks the stack to show where that ceiling sits and why it holds at every level.

The orchestrators: built for a different shape of work

Start at the foundation. SLURM emerged from scientific computing around 2003, optimized for fixed clusters running long batch jobs. Kubernetes descends from a lineage built for web services. Both predate modern AI by two decades, and both were designed around an assumption that no longer holds: that a job is a thing you place on a machine and leave there.¹

The friction shows up immediately when AI work meets that assumption. Vanilla Kubernetes has no native concept of gang scheduling, so a multi-node training job can deadlock with three pods running and the fourth stuck pending because no node has free GPUs.¹² The ecosystem has responded with real engineering: Dynamic Resource Allocation graduated to general availability in Kubernetes 1.34, replacing the old integer-count device model with attribute-based GPU requests evaluated natively by the scheduler.³ None of this is trivial and none of it is bad work. But notice what all of it does. It improves how a pod is matched to a node. It is placement, refined.

The next-generation schedulers: placement, refined further

A whole tier of more sophisticated schedulers has grown up to close the gap Kubernetes left. Volcano, YuniKorn, and Kueue add gang scheduling, fair-share, and multi-cluster dispatch, with Volcano reporting two-to-four-times scheduling performance over the default.²⁴ GPU-topology-aware schedulers coordinate bin-packing and fair-share allocation with awareness of the network fabric. Run:ai and similar platforms add fractional GPU sharing so a card can be split across jobs.³⁴

These are real advances, and fractional sharing in particular gets closer to the problem than anything below it, because it admits that one job rarely needs a whole GPU. But the unit is still the unit. Fractional sharing slices the hardware to fit the job; it does not change the job to fit the hardware. The scheduler still receives a workload, decides where it goes, and hands it over intact. The HPC and cloud-native worlds are even converging, with Slurm now running on Kubernetes through projects like Slinky and SUNK so teams get batch semantics on a cloud-native control plane.⁵⁶ Convergence at the placement layer is the headline of 2026. It is also, precisely, convergence at the placement layer.

The serving frameworks: hard engineering, above the unit

The inference-serving layer is where the placement engineering goes deepest, and it is worth being precise about what it does. vLLM's PagedAttention treats the key-value cache like virtual memory, breaking it into non-contiguous blocks to reclaim the 60 to 80 percent of memory that fragmentation used to waste, and continuous batching lets new requests join an in-flight batch the moment a slot opens.⁷ SGLang, TensorRT-LLM, and the rest share these primitives and compete on the margins of throughput and latency.⁸⁹

Look at how these systems describe their own architecture, though, and the pattern is plain. Production serving engines run a global router that distributes requests across engine replicas, and each engine schedules requests internally into dynamic batches at iteration granularity.¹⁰¹¹ Every word of that is scheduling. Request distribution, batch formation, iteration-level admission and retirement, KV-cache management: all of it is placement and memory management around a unit of work that arrives fixed and leaves fixed. The framework makes the GPU busier serving the request. It does not change what the request is made of or where its stages run.

Disaggregation: the field reaches the edge of placement

The most advanced answer the industry has shipped is disaggregation, and it gets closest to the actual problem. NVIDIA's Dynamo went to general availability at GTC in March 2026, splitting the prefill and decode phases of inference onto dedicated hardware pools because co-locating them on one GPU wastes capacity, with NVIDIA reporting up to roughly 7x throughput on Blackwell from the separation. Google Cloud publishes a recipe on the same principle, and Gartner, Anyscale, Microsoft, and the FinOps Foundation describe the fix the same way.¹²¹³

This is the field arriving, independently and at scale, at the right diagnosis: stop running one undifferentiated unit on one resource, and separate the stages. It is the strongest validation possible that the unit of work is the problem. But look at where it stops. Dynamo is explicitly an orchestration layer that routes between worker pools on top of Kubernetes.¹² It disaggregates one workload, inference, into two phases, prefill and decode, and it does that placement inside the existing paradigm. It is the most sophisticated placement engine yet built, and it has reached the edge of what placement can do: it separates two known stages of one known workload and routes them well. It does not change the fundamental unit, and it does not generalize beyond the case it was built for.

Why the ceiling holds

Stack the layers and the ceiling is visible as a single line. Orchestrators place jobs on nodes. Next-gen schedulers place them better and slice the hardware finer. Serving frameworks place requests into batches and manage memory around them. Disaggregation places two phases of one workload onto two pools. Every layer improves the answer to "where and when." None of them changes the answer to "what."

That is why the gap stays open. The waste documented across this series, average GPU utilization near 5 percent, is created at the layer where work meets hardware, inside the unit, before any scheduler sees it.¹⁴ A workload that bundles CPU and GPU stages into one container will scale as one unit no matter how perfectly it is placed, because the mismatch is in the unit, not the placement.¹⁵ You cannot place your way out of a problem that lives inside the thing being placed. This is not a knock on schedulers. It is a statement about which layer the waste lives in, and the waste does not live in the scheduling layer.

Where the other layer is

There is a second pile, and it has almost nothing in it, because changing the unit of work requires standing in a specific and largely empty place: below the orchestrator, above the silicon, at the layer where work actually meets hardware.

That is where TAHO operates, and it is what makes the position complementary rather than competitive with everything described above. TAHO does not replace the scheduler, the serving framework, or the disaggregation engine. It changes what they are scheduling. It breaks a workload into smaller units, sends each unit to the resource best suited to run it, and runs it there, on any silicon, beneath all the placement intelligence the rest of the stack has spent two decades building. The orchestrator keeps issuing intent. The serving framework keeps batching. Dynamo keeps separating prefill and decode. All of it gets better, because the thing underneath them finally fits the work to the metal before they ever place it.

The point

From SLURM to Dynamo, the history of AI infrastructure is a history of placement getting smarter, and that progress is real and worth respecting. But it has reached its own edge. Every layer in the stack takes the unit of work as given, and the waste is created inside the unit, which is why more and better placement has not closed the gap and structurally cannot.

The whole field has agreed on the destination: break execution apart and run each stage on the resource it needs. The layers above the unit have taken that idea as far as placement allows. Reaching it fully means operating on the unit itself, in the one layer beneath all of them. That layer is not a better scheduler. It is what the scheduler has been waiting for underneath it.

A note on the numbers

The capability claims here are drawn from primary and vendor sources current to Q1 to Q2 2026, and each tool is described by what it actually does rather than by a competitive caricature. Performance figures (vLLM's 60 to 80 percent memory reclamation, Volcano's 2x to 4x scheduling improvement, Dynamo's ~7x throughput) are vendor- or benchmark-reported on specific configurations and labeled as such; they are cited to credit the engineering, not to rank the tools. The 5 percent utilization figure is the same direct-measurement number used across this series, reported with what it measures. The architectural claim, that every layer operates on placement rather than on the unit of work, is the load-bearing argument, and it rests on how these systems describe themselves, not on any single performance number. As with the execution gap, the argument rests on the shape of all of them together.

References

1. SLURM originating in scientific computing around 2003 for fixed clusters and long batch jobs; Kubernetes descending from a web-services lineage; neither designed for modern AI's shape, with vanilla Kubernetes lacking native gang scheduling and deadlocking multi-node jobs. SkyPilot (2025); Spheron (May 2026). SkyPilot Spheron
2. Kubernetes default scheduler lacking gang scheduling, causing GPU deadlocks; Volcano, YuniKorn, and Kueue adding batch scheduling, with Volcano delivering 2x-4x scheduling performance over default and Kueue enabling multi-cluster dispatch, while inheriting Kubernetes complexity. SkyPilot. SkyPilot
3. Dynamic Resource Allocation (DRA) graduating to GA in Kubernetes 1.34, replacing the integer-count device model with attribute-based GPU requests; GPU-topology-aware scheduling and fractional GPU sharing via NVIDIA Run:ai. CloudOptimo (2026). CloudOptimo
4. Next-generation schedulers (Volcano, YuniKorn, Kueue) and Run:ai adding gang scheduling, fair-share, fractional GPU sharing, and topology awareness as additions to the placement layer. SkyPilot; CloudOptimo. SkyPilot CloudOptimo
5. Slurm-on-Kubernetes convergence via Slinky (SchedMD, now part of NVIDIA), bringing Slurm scheduling to Kubernetes pods and running full Slurm clusters as pods with topology-aware multinode scheduling. NVIDIA Technical Blog (May 2026). NVIDIA Developer
6. SUNK (Slurm on Kubernetes) combining Slurm batch semantics with the Kubernetes control plane; HPC and cloud-native scheduling converging toward a single control plane. Online Inference / Medium (Mar 2026); CloudOptimo. Medium CloudOptimo
7. vLLM PagedAttention treating the KV cache like virtual memory to reclaim the 60-80% of memory lost to fragmentation, plus continuous batching admitting new requests into an in-flight batch. Red Buffer / Medium (Apr 2026). Medium
8. vLLM, SGLang, and TensorRT-LLM sharing core optimizations (continuous batching, paged KV cache, prefix caching, speculative decoding) and competing on performance, hardware breadth, and developer experience. Inference Engineering (2026). Inference Engineering
9. SGLang focused on structured, multi-step generation and controlling how generation unfolds, versus vLLM's throughput focus; teams often mixing approaches. Yotta Labs (Feb 2026). Yotta Labs
10. Production serving systems (vLLM, SGLang, TGI) adopting a layered architecture of a global router plus inference engines, with scheduling at engine level (across replicas) and request level (dynamic batching). arXiv 2602.21626. arXiv
11. Iteration-level scheduling with continuous batching as the default control primitive across production engines; complementary schedulers (Sarathi-Serve chunked prefill, Fast-Serve token-granular preemption) operating at the same placement layer. arXiv 2601.17855. arXiv
12. NVIDIA Dynamo at GA (GTC, March 16, 2026) splitting prefill and decode onto dedicated pools with up to ~7x on Blackwell; an orchestration layer routing between worker pools on Kubernetes. NVIDIA Spheron
13. Google Cloud disaggregated-inference recipe; convergence on stage separation across Gartner, Anyscale, Microsoft, and the FinOps Foundation. Google Cloud Gartner
14. Cast AI, 2026 State of Kubernetes Optimization Report. Average GPU utilization of 5% across tens of thousands of production clusters; utilization going backward where orchestration matured. Cast AI SDxCentral
15. Anyscale, GPU (In)efficiency in AI Workloads. Single-container packaging forces CPU and GPU stages to scale as one unit, guaranteeing low utilization regardless of how well the workload is placed. Anyscale