The gap lives at execution, so that is where you close it
The gap is not a scheduling problem. Use has not improved as orchestration has matured. In the environments where orchestration became the default foundation for AI, measured use went backwards.³⁴ More scheduling on top of the same execution model manages the gap. It does not close it.
The reason is structural. Orchestration decides where work is placed and when it moves. It does not change the unit of work itself, and it does not change the path that work takes to reach the resource best able to run it. Scheduling intent lives above. Realized utilization happens below, at execution, on the metal. The waste is created at the layer where work meets hardware, and that is the only layer where it can be recovered.
So we build there. Below the orchestrator. Above the silicon. The exact layer where lost compute is lost.
Why not higher, why not lower
Build higher, inside orchestration, and you inherit its ceiling. You can place work better, reserve headroom more carefully, pack containers more tightly, and you still hit the same wall, because the unit of work and its path to the resource are fixed before you ever get to schedule it. Single-container packaging forces CPU and GPU stages to scale as one unit, which guarantees low utilization no matter how well you schedule.⁵ You are optimizing the symptom.
Build lower, in the silicon, and you are NVIDIA. That is a different company and a closed one. You would be betting on a single vendor's hardware in a world that is getting more heterogeneous, not less. Fleets are already a mix of GPU generations, accelerators, and other silicon spread across clouds, regions, and edge. Owning one kind of chip does not help you run work well across all of them.
The layer between is the only neutral control point. It is high enough to be hardware agnostic and low enough to change what actually executes. You do not compete with the model above. You do not compete with the chip below. You change what the scheduler is scheduling.
What standing here lets us do
From this layer, TAHO changes the unit of execution. We break a workload into smaller units, send each unit to the resource best suited to run it, and run it there. The thesis is simple to say and hard to build. Capacity is no longer the constraint. Fit is.
Three things follow from the position itself.
We abstract the silicon without being the silicon. We present one execution fabric across mixed hardware, so everything above stops caring what is underneath. The messier the fleet gets, the more that abstraction is worth. Heterogeneity is the moat, not the problem.
We capture the economics where they are won. Scheduling intent is decided above us. Realized utilization happens at our layer. Average GPU use across tens of thousands of production clusters sits near 5 percent, which means organizations are paying for roughly twenty times the capacity they actually use.³⁶ Every point of utilization recovered is direct return on the most expensive asset the customer owns. That is a CFO-legible number, which is rare for infrastructure.
We remove the central bottleneck. Our decentralized execution fabric places and runs work peer to peer rather than through a single control plane. That is exactly where centralized orchestration breaks down at scale, and it is a property of where we sit, not a feature bolted on top.
Complement now
We do not ask anyone to rip anything out. TAHO sits beneath the orchestrator and above the hardware and makes the orchestrator's decisions execute better. The orchestrator keeps issuing intent. We change how that intent meets the metal.
That makes the first yes cheap. There is no migration, no platform fight, no displacement of the team that owns the scheduler. The entry case is utilization recovered on hardware the customer already paid for, measured in their own numbers. The rest of the field is arriving at the same diagnosis. Vendors and analysts now describe the fix the same way we do: break execution apart and run each stage on the resource it actually needs.⁷⁸⁹ This is no longer theory. NVIDIA's own disaggregated inference framework went to general availability in March 2026, shipping prefill-decode separation as production software rather than a research idea.⁷ The direction of the field is settled. We were built to do that. Standing where we stand is what lets us do it without asking the customer to bet the farm to find out.
The point
Capacity stopped being the constraint. Fit became the constraint. The gap that hides the difference lives at execution, below the orchestrator and above the silicon, and that is the one place it can be closed. So that is where we build.
A note on the numbers
The figures here are reported with their source and what they measure. Peak-compute figures are dense FP16 from primary NVIDIA datasheets,¹² held to a consistent precision so generations compare cleanly. Utilization claims distinguish fleet utilization, model-FLOPs utilization, and server-capacity utilization, because they are three different things. The disaggregation evidence is drawn from the primary sources closest to the hardware, including NVIDIA and the analysts who reached the same conclusion independently. As with the execution gap, the argument does not rest on any single number. It rests on the shape of all of them together.
References
- NVIDIA H100 Tensor Core GPU Datasheet. FP16 dense 989 TFLOPS; the headline "2,000 TFLOPS*" figure is the sparse number. NVIDIA
- GB300 NVL72 architecture and per-precision dense figures (B200 2,250 vs GB300 2,500 dense FP16/BF16, +11.1%). Verda; V100 dense FP16 of 125 TFLOPS corroborated at Spheron. Verda Spheron
- Cast AI, 2026 State of Kubernetes Optimization Report. Average GPU utilization of 5 percent across tens of thousands of production clusters (AWS, Azure, GCP); CPU and GPU both down year over year. Cast AI Cast AI
- SDxCentral, "Kubernetes efficiency is going backwards as AI drives GPU waste." Documents the year-over-year decline. SDxCentral
- Anyscale, GPU (In)efficiency in AI Workloads. Single-container packaging forces CPU and GPU stages to scale as one unit, guaranteeing low utilization; argues for disaggregated, multi-stage execution. Anyscale
- Independent reporting on the Cast AI findings: ~5 percent average across ~23,000 clusters, roughly 20x over-allocation. ITBrief. ITBrief
- NVIDIA, introducing Dynamo. Co-locating prefill and decode on one GPU leads to inefficient resource use; disaggregating the phases lets each be optimized independently. Dynamo reached general availability (1.0) at GTC on March 16, 2026. NVIDIA Developer NVIDIA
- Gartner, "Build Strategic Differentiations for On-Premises AI Infrastructure Offerings" (doc 7211630, November 21, 2025). Recommends shared GPU usage across siloed projects plus prefill-decode disaggregation with heterogeneous processors. Gartner Computer Weekly
- VentureBeat, coverage of the cross-vendor convergence (Cast AI, Anyscale, Gartner) toward disaggregated inference. Yotta Labs on production LLM inference commonly observed at 20 to 40 percent GPU utilization. VentureBeat Yotta Labs