In 2026, the world's nine largest cloud providers will spend roughly $830 billion on data center infrastructure¹:
Microsoft alone is on track for $190 billion
Google between $180 and $190 billion
AWS will pass $230 billion
Meta is approaching $145 billion¹
Inside the buildings that money is paying for, average GPU utilization sits at a truly shocking 5%.²
The number comes from Cast AI's 2026 State of Kubernetes Optimization Report,² drawn from telemetry across tens of thousands of production Kubernetes clusters -- the default state for most enterprise AI infrastructure, not a frontier-lab outlier. It's an average, which means a meaningful share of fleets run below it.
"A GPU sitting idle costs dollars per hour. A CPU sitting idle costs cents. And 95% of GPU capacity is doing nothing. Cloud vendors just raised H200 prices 15%, breaking a 20-year trend of falling compute costs. That's not a configuration problem as much as it is a business emergency."
-- Laurent Gil, co-founder and president, Cast AI²
The industry's response has been to build more. Hyperscaler 2027 capex is projected at roughly $1.1 trillion,³ and VentureBeat recently characterized the underlying utilization gap as "the $401 billion AI infrastructure problem enterprises can't keep ignoring."¹⁹ Spending $1.1 trillion to chase a utilization problem is one approach. Fixing the utilization problem is another.
The floor and the ceiling
The Cast AI study averages production fleets running real AI workloads, not lab benchmarks. Many clusters sit in the low single digits. A handful run higher -- though not as much higher as the press releases imply.
SemiAnalysis, an independent research and analysis company specializing in the Semiconductor and AI industries, places the well-tuned production ceiling at 40-55% Model FLOPs Utilization (MFU) -- a level reached only after months of dedicated tuning:¹⁶
Meta's published Llama 3 training -- one of the largest, most heavily engineered runs of the era -- hit 38-43% BF16 MFU at scale¹⁷
ByteDance reported 55.2% MFU training a 175B-parameter model on 12,288 GPUs, among the highest production numbers on public record¹⁸
The well-tuned ceiling tops out near 55%. Most enterprise fleets live closer to the 5% average. It proves that even the best operators in the world leave nearly half the hardware idle.
Sources: Cast AI · SemiAnalysis · Llama 3 paper (Meta) · MegaScale paper (ByteDance)
The capacity is there. Today's orchestration wasn't built to reach it.
The hardware in modern AI fleets is the most computationally dense silicon ever shipped. In recent industry benchmarks, AMD's flagship cluster posted 90%+ scaling efficiency across multiple nodes.⁷ When workload, topology, and orchestration are aligned, modern accelerators saturate cleanly. The 5% comes from what happens between those benchmark runs.
Three architectural assumptions drive most of the gap.
The first is centralized scheduling. Today's orchestration patterns were designed for long-lived services, batch jobs, and microservice traffic -- workloads a single scheduler could plan across the cluster. AI workloads don't behave that way. They have bursty fan-out, heterogeneous resource shapes, and short tasks better claimed in microseconds than scheduled in seconds. A central planner becomes the bottleneck the moment work outpaces its planning rate.
The second is coarse decomposition. Most schedulers reason about machines or pods; AI workloads decompose well below that, to the thread, kernel, or token level. Assign work at machine granularity and every machine you assign to has dozens of execution units idle while the assigned work runs.
The third is the absence of a deduplication layer. Inference and agentic workloads run the same computations repeatedly: identical prompts across users, recomputed intermediate states across sessions, retrieval chains that vary only at the margins. Without content-addressing of computation, that work runs from scratch every time. Cloudflare's recent move to a disaggregated prefill-decode architecture pushed its input-token cache hit ratio from 60 to 80%⁸ -- one change that nearly halved the cost of redundant work on a single workload type. Most of the industry is still doing the work twice.
These are software problems showing up as hardware bills.
The cost story
Falling compute costs have been one of the more reliable trends in modern technology. From the late 1990s through the mid-2020s, the price per unit of compute declined steadily, decade over decade. In 2026, that trend broke. Cast AI reports cloud vendors raised H200 prices 15% during the year, the first sustained increase in two decades.²
Writing in VentureBeat, analyst Rob Strechay frames the gap as a finance problem rather than an engineering one.
"For every dollar spent on silicon, 95 cents is essentially a donation to a cloud provider's bottom line. In any other department, a 95% waste metric would be a firing offense. In AI infrastructure, it's just called 'preparedness.'"
-- Rob Strechay, contributing analyst, VentureBeat¹⁹
Strechay notes that for enterprises like Intuit, Mastercard, and Pfizer, GPU access was rarely the actual bottleneck. Those companies secured capacity that then sat idle while internal teams worked through data gravity, governance, and architectural immaturity.¹⁹
The financial pressure is everywhere, and not subtle. OpenAI's CFO Sarah Friar has reportedly expressed concern that the company can't afford its future compute contracts if revenue growth doesn't catch up, according to Wall Street Journal reporting.⁹ Even Microsoft is reportedly weighing whether to walk back its 2030 clean energy goal -- not because the goal stopped mattering, but because the buildout has made it impossible to hit.¹¹
Translation: the largest balance sheets in tech are restructuring their plans around a buildout they can't fully afford.
Every previous compute era followed the same pattern: usage explodes, unit cost falls. AI broke the bull case on unit economics this year. And yet there was no press release.
Provider
2026 Capex (USD billions)
AWS
$230B
Microsoft
$190B
Google
$185B
Meta
$135B
Oracle
$35B
ByteDance
$30B
Tencent
$15B
Alibaba
$8B
Baidu
$2B
Source: TrendForce, May 2026
A 5%-utilization fleet running on hardware 15% more expensive year over year doesn't survive contact with normal market discipline. The companies making that bet are betting demand keeps absorbing the cost. That works until it doesn't.
The supply side perfect storm
The instinct to build more is running into a convergence of yet more challenges: power, specialty supply chain, and pipeline.
On power: even where capacity exists on paper, getting it onto the US grid has become its own bottleneck.
In Nevada, Fleet Data Centers filed in April to build behind-the-meter natural gas generation because NV Energy can't serve the site for at least two years.¹²
North Carolina is debating a bill that would force 40+ MW data centers onto full-cost-recovery rates and require 25% on-site clean generation.¹³
The cost to build a new combined-cycle gas turbine plant has risen 66% over three years to $2,157 per kilowatt, with construction taking 23% longer; equipment waitlists now stretch into the early 2030s.⁵
On the specialty supply chain: NVIDIA preemptively secured large volumes of 4nm and 3nm wafer capacity, CoWoS packaging, T-glass substrates, PCBs, HBM, and SSDs. Peers including Google now face material shortages constraining product growth.¹⁴
On the pipeline itself: Sightline Climate's Q1 2026 outlook tracks 190 GW across 777 announced data center projects since 2024. Only 5 GW of the 16 GW slated for 2026 is actually under construction. The firm estimates 30 to 50% of the 2026 pipeline will not come online this year.⁶
The honest case for building more anyway runs like this: demand will close the gap, supply chains will catch up, and unit economics ease on the far side of scale. That argument worked in every previous compute era. The problem in 2026 is that two of those three assumptions -- elastic supply, economics improving at scale -- have already broken on the data published this year.
The "build more" thesis worked when the bottleneck was demand and supply was elastic. Neither condition holds anymore.
If you can't buy your way out, fix what you already own
The last two sections made the negative case: the cost curve has flipped, and the supply chain can't deliver on schedule. The positive case sits one layer up. The 5% utilization gap isn't a hardware problem -- it's a software problem, and it lives in the execution fabric beneath orchestration. The unlock isn't on the bill of materials.
Invert each of the three architectural assumptions and the economics flip. Move coordination beneath the orchestrator so machines discover and claim work themselves, and idle hardware finds tasks in microseconds instead of waiting for the next planning cycle. Decompose work below the machine boundary, and the dozens of execution units inside each accelerator stop being invisible to the scheduler. Content-address computation, and redundant work becomes a cache miss rather than a CPU cycle -- coverage compounds as patterns repeat, so the economics improve with use.
None of this requires new hardware. It requires a different relationship between software and the machines underneath it.
Enterprise buying behavior is already turning in this direction. Strechay's Q1 2026 AI Infrastructure and Compute Market Tracker shows GPU availability as a deciding factor in provider selection falling from 20.8% to 15.4% in a single quarter, while cost per inference and TCO climbed from 34% to 41% of buyer priorities. Specialized AI cloud adoption rose from 30.2% to 35.9% in the same window.¹⁹ The procurement question has moved from "can I get more GPUs" to "am I using the ones I already have."
The companies that prevail through the next phase will not be the ones that bought the most GPUs. They will be the ones whose GPUs were doing something other than quietly depreciating in a rack.
The CPU pivot underneath
There's a quieter signal in the same dataset, and it's the one most of the industry isn't looking at.
AMD reported in early May 2026 that AI infrastructure's CPU-to-GPU ratio is shifting in a direction nobody priced in.¹⁵ In chatbot workloads, the ratio sat at 1:4 to 1:8. In agentic workloads, it's moving toward 1:1 -- and in some configurations, tipping toward CPU-heavy. AMD raised its server CPU TAM forecast from 18% to over 35% annual growth, projecting more than $120 billion by 2030.
That changes an assumption that has driven AI infrastructure conversations for three years -- that the only compute that matters is GPU compute. Agentic workloads (chains of reasoning, tool calls, retrieval, evaluation) look more like traditional services: bursty, heterogeneous, latency-sensitive. Many of those components run faster on saturated CPUs than on underutilized GPUs. The infrastructure response is not "buy more GPUs." It's "use the CPUs you already own."
The 5% number, the cost trend, the supply ceiling, and the agentic CPU pivot all carry the same message. The next phase of AI infrastructure value comes from getting more out of the machines that already exist -- not from buying more of them to compensate for the ones that aren't earning their keep.
Where this leaves us
Every previous compute era had a visible inflection -- GPU acceleration was the most recent, cloud before that, containerized deployments before that. The headline was hardware in every case, but the change underneath was in what the software running on top assumed about the hardware below.
The 2020s build-out has so far been a hardware story: $830 billion in 2026, more than a trillion projected for 2027. The 5% utilization average suggests the next chapter is different. The hardware already deployed is mostly idle. The hardware coming in the next two years will hit a supply ceiling before it hits the floor. Buying out of the problem stops working somewhere short of where demand is going -- a fact the CFOs writing these checks can already see on their own spreadsheets.
The compute we have isn't being used. The compute we're building won't all arrive. The companies that operate on both facts at once will be in a different market than the ones still buying their way out of the bottleneck.
References
TrendForce, "North American AI Data Center Expansion Drives 2026 Capex of Top Nine CSPs to US$830 Billion," May 2026. TrendForce
Cast AI, "2026 State of Kubernetes Optimization Report." Cast AI · Independently reported by ITBrief: ITBrief
Morgan Stanley hyperscaler capex forecast, May 2026. Crypto Briefing
Energinet, "Temporary pause on new grid connection agreements," May 2026. Energinet
BloombergNEF, "Data center demand drives 66% surge in natural gas power plant costs," April 2026. TechCrunch
AMD, "Instinct MI355X MLPerf Inference v6.0 results," May 2026. AMD
Cloudflare, "Disaggregated prefill-decode architecture for Workers AI," April 2026. Cloudflare
"OpenAI Misses Key Revenue, User Targets in High-Stakes Sprint Toward IPO," The Wall Street Journal, April 2026. WSJ
Tom's Hardware, "Mark Zuckerberg says Meta is cutting 8,000 jobs to pay for AI infrastructure," May 2026. Tom's Hardware
TechCrunch reporting on Bloomberg, "Microsoft's AI data center push is colliding with its clean power goals," May 2026. TechCrunch
The Nevada Independent, "Why a northern Nevada data center wants to build its own temporary natural gas power plant," April 2026. The Nevada Independent
Data Center Knowledge, "North Carolina targets hyperscale costs with proposed AI infrastructure bill." Data Center Knowledge
TrendForce, "AI demand tightens advanced packaging and wafer capacity," April 2026. TrendForce
AMD, "Agentic AI changes the CPU-GPU equation," May 2026. AMD
SemiAnalysis, "H100 vs GB200 NVL72 Training Benchmarks: Power, TCO, and Reliability Analysis, Software Improvement Over Time." SemiAnalysis
Meta AI, "The Llama 3 Herd of Models," arXiv 2407.21783, 2024. arXiv
ByteDance, "MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs," arXiv 2402.15627, 2024. arXiv
Rob Strechay, "5% GPU utilization: The $401 billion AI infrastructure problem enterprises can't keep ignoring," VentureBeat, May 2026. VentureBeat
Get new Insights.
New articles from TAHO, delivered to your inbox. No spam.
