The constraint moved
For most of the cloud era, the constraint on compute-heavy businesses has been capacity. You needed more, you bought more, and the only real questions were budget and lead time. That world is ending in front of us, for two reasons that point the same direction.
The first is that the industry is buying capacity at a scale that has no precedent and cannot continue forever. The five largest US cloud and AI companies are guiding toward roughly $635 to $690 billion in combined 2026 capex, more than double 2024 levels, with about three quarters of it going to AI infrastructure.¹ Goldman Sachs models roughly $7.6 trillion in cumulative AI-related capex between 2026 and 2031 across compute, data centers, and power.² Capex is now growing faster than the cloud revenue meant to justify it, and some hyperscalers are dedicating a historically extraordinary share of revenue to infrastructure.³ This is a supply curve being pushed vertical by competitive fear, not by unit economics.
The second is that the input underneath that spend is no longer scarce in the way it was. H100 cloud rates have fallen 64 to 75 percent from their peak, and used H100s now move on the secondary market for a fraction of their 2023 price. The same hardware that was rationed eighteen months ago is now available across forty-plus providers at a 20x-plus spread between the cheapest and most expensive listing.⁴ Raw access to accelerators is commoditizing.
When the thing everyone was racing to acquire becomes abundant and cheap, owning more of it stops being a moat. The constraint moves somewhere else. It has moved to fit: how well a given unit of work is matched to the resource that runs it.
Why the constraint is now physical, not financial
There is a tempting counterargument: if capex keeps climbing, capacity is clearly still the game. The opposite is true, and the reason is that the build-out has hit a wall money cannot climb.
Every major hyperscaler now reports that new capacity is gated by grid interconnect timelines, transformer lead times measured in 18 to 24 months, and permitting, not by willingness to spend. Microsoft has disclosed an Azure backlog it cannot fulfill because it lacks the power to bring capacity online. AI data center power demand is projected to reach 156 GW by 2030.⁵ A company can sign a purchase order for GPUs in an afternoon. It cannot buy a substation, an energization date, or a jurisdiction willing to absorb concentrated load on the same timeline.
This is the hinge of the thesis. When the constraint was financial, the answer was always to buy more. Now that the constraint is physical, that answer is foreclosed. You cannot out-spend a transformer shortage. The only lever left is to extract more useful work from the capacity that is already powered, cooled, and racked. That lever is fit.
What fit is worth
The prize is hiding in plain sight, and it is large because the waste is large.
Across tens of thousands of production clusters, average GPU utilization sits near 5 percent, which means organizations are paying for roughly twenty times the capacity they actually use.⁶⁷ Even well-tuned training runs reach only 35 to 45 percent of theoretical peak. Production inference typically runs 20 to 40 percent before optimization.⁸⁹ The gap between what is bought and what is used is not a rounding error. It is the majority of the asset.
Put a price on it. An H100 on a major hyperscaler runs roughly $7 to $12 per GPU-hour.⁴ At 5 percent utilization, the buyer is paying full freight for an asset doing one-twentieth of its possible work. Closing even part of that gap is not an incremental optimization. It is the difference between needing a second data center and not needing it, between a negative free-cash-flow year and a positive one. In a world where the next megawatt is 24 months out, recovered utilization is the only capacity available on demand.
This is why fit is the constraint that matters. It is the one input that is both enormous in magnitude and immediately addressable, while every other lever is gated by physics and permitting.
The whole industry has converged on the same answer
The strongest evidence that fit is the real game is that the people closest to the hardware are now saying so, in their own words, with their own products.
The pattern is the same everywhere: stop running undifferentiated work on undifferentiated hardware, and instead break execution into stages that each run on the resource they actually need. NVIDIA shipped Dynamo to general availability at GTC 2026 precisely to split the prefill and decode phases of inference across dedicated hardware, because co-locating them on one GPU wastes capacity. NVIDIA reports up to roughly 7x throughput gains from this separation on Blackwell.¹⁰ Google Cloud now publishes a disaggregated-inference recipe built on the same principle.¹¹ Gartner, Anyscale, Microsoft, and the FinOps Foundation have all arrived at the same diagnosis from different angles.⁸¹²
This convergence is the validation and the risk in one fact. The validation: the entire field now agrees that fit, not capacity, is where the value is. The risk: everyone is reaching for the same prize, which means the question for any investor is not whether fit matters but who closes the gap most completely.
Where the current answers stop
The industry's answers so far share a ceiling. They disaggregate one workload, inference, into two phases, prefill and decode, and they do it inside the orchestration paradigm that already exists. Dynamo is explicitly an orchestration layer that routes between worker pools on top of Kubernetes.¹⁰ It decides where work is placed and when it moves. It does not change the fundamental unit of work, and it does not change the path that work takes to reach the resource best able to run it.
That is why more scheduling on top of the same execution model manages the gap rather than closing it. Utilization has gone backward in exactly the environments where orchestration matured.⁶¹³ You cannot fix a unit-of-work problem with a placement engine, because placement takes the unit as given.
Fit, fully realized, requires changing the unit itself: breaking a workload into smaller units, sending each to the resource best suited to run it, and running it there, in a layer beneath the scheduler and above the hardware. Not a replacement for orchestration. A change in what the scheduler is scheduling.
The fit thesis
Three facts, taken together, define the opportunity.
Capacity is commoditizing and physically capped, so buying more is neither a moat nor, soon, an option. The waste sitting inside already-powered capacity is the majority of the asset, and it is the only capacity available without a 24-month wait. And the entire industry has now publicly agreed that the fix is to fit work to the resource that runs it, while shipping only partial answers that operate above the unit of work rather than on it.
The teams that fit work to machines will get more out of the hardware they already pay for, spend less to deliver the same result, and begin to compete on how well they execute rather than how much they spend. That efficiency is the value the build-out has been hiding. Capacity was the last decade's constraint. Fit is this one's.
A note on the numbers
Pricing and capex figures here are drawn from primary and analyst sources current to Q1–Q2 2026 and reported with the spread intact rather than the most favorable point, because the spread is the story: a 20x range in GPU rates and a 20x gap between bought and used capacity describe the same mismatch from two sides. Utilization figures are reported with what they measure, since fleet utilization, model-FLOPs utilization, and server-capacity utilization are three different things. The Rubin-era and 2030 projections are labeled as projections. As with the execution gap, the argument does not rest on any single number. It rests on the shape of all of them together.
References
- Hyperscaler 2026 capex of roughly $635–690B, about three quarters to AI infrastructure. LongYield (Mar 2026); corroborated by Futurum (~$690B) and Introl (>$600B, 36% YoY). LongYield Futurum Introl
- Goldman Sachs Global Institute (May 2026), roughly $7.6T cumulative AI-related capex 2026–2031. Goldman Sachs
- Capex outpacing cloud revenue; historically high share of revenue directed to infrastructure. LongYield; invezz via TradingView. LongYield TradingView
- H100 cloud rates down 64–75% from peak; used-card collapse; 20x+ provider spread; hyperscaler on-demand ~$6.88 AWS / ~$12.29 Azure. CloudZero (Q1 2026), Spheron (14 May 2026), getdeploying (45+ providers). CloudZero Spheron getdeploying
- Power as the binding constraint; interconnect 18–36mo, transformers 18–24mo; Microsoft Azure backlog; 156 GW by 2030. Futurum, AL Capital Advisory (CFA analysis, May 2026), LongYield. Futurum AL Capital Advisory LongYield
- Cast AI, 2026 State of Kubernetes Optimization Report. Average GPU utilization of 5% measured across tens of thousands of production clusters (AWS, Azure, GCP); both CPU and GPU down year over year. Cast AI
- Independent reporting on the Cast AI findings: ~5% average across ~23,000 clusters, roughly 20x over-allocation. ITBrief. ITBrief
- Production LLM inference commonly at 20–40% GPU utilization. Yotta Labs. VentureBeat coverage of the cross-vendor convergence (Cast AI, Anyscale, Gartner) toward disaggregated inference. Yotta Labs VentureBeat
- 35–45% MFU on well-tuned training runs. Meta Llama 3 (arXiv 2407.21783); CoreWeave H100 benchmarks. arXiv CoreWeave
- NVIDIA Dynamo reached general availability (1.0) at GTC on March 16, 2026, shipping prefill/decode disaggregation as production software with up to ~7x throughput on Blackwell; an orchestration layer routing between worker pools on Kubernetes. NVIDIA Spheron Digital Applied
- Google Cloud disaggregated-inference recipe using NVIDIA Dynamo on AI Hypercomputer. Google Cloud
- Convergence on stage separation across Anyscale, Gartner, and the FinOps Foundation. Anyscale Gartner FinOps Foundation
- SDxCentral, "Kubernetes efficiency is going backwards as AI drives GPU waste." Documents the year-over-year decline alongside the Cast AI data. SDxCentral