Most businesses pay for AI compute against a meter that runs whether or not any work gets done, realizing value only when work finishes. At current average utilization rates, there's a twenty-to-one gap: the meter charges for twenty units, the work returns one. Closing that gap is the one line in the budget that improves without buying anything new or switching vendors, because the money is already committed. Every other way to cut compute costs asks you to spend or to move. So why not use what you've already paid for?
You pay by the clock, you earn by the work
A microchip is an expensive specialist that does the heavy AI math; read "chip" here as an expensive accelerator ("GPU"), any brand. Time, not output, sets its true cost. Rented, it bills by the hour the moment it's reserved, running or not: ~$2 to $3.50 an hour from specialist providers, several times that on the big clouds.¹² Purchased outright, it costs $25,000-$40,000 up front, losing value by the day and as newer chips arrive, whether or not it ever ran.³ Either way, the meter is tied to time held, not to work delivered.
Value realization runs on a completely different clock. A business gets something back only when a unit of work completes: a model trained, a query answered. Whether or not full value is realized against the committed spend comes down to whether or not the "paid-for compute" and "work value" clocks stay close together.
Guess what?
They've come apart.
Measured across tens of thousands of production systems, some of the most expensive chips do useful work about 5% of the time.⁴⁵ The billing clock runs for the whole hour, the work clock for about three minutes. That's not a small or infrequent gap. It's roughly 20:1, and it's the default state of a typical compute fleet.
Burning money you've already spent
The twenty-to-one gap isn't a future cost to avoid or a purchase to weigh. It's sunk cost; money that's already left the account. Whether the chip is rented or bought, running or not, the full bill is paid. The key question is how much work came back for that spend, and the answer is about a twentieth of what you paid for.
Most cost levers in an infrastructure budget require doing something expensive to capture them: negotiate a new contract, commit to a longer term, migrate to a cheaper provider, buy more efficient hardware. Each lever costs money, time, or both before it pays back. The gap between the "paid-for compute" and "work value" clocks is entirely different. Capacity is already bought & paid for. Yielding more value doesn't require a new hardware purchase, a new service contract, a platform migration, or a new supplier. It requires getting work onto hardware that's already sitting there, meter running.
This gap isn't the price of running AI, though many assume so as the bill climbs, shrugging it off as the cost of playing in the field. The great irony is that it's the cheapest money in the budget to recover, because recovering it is the only move on the table that doesn't start with spending more.
Why the gap exists, why it's recoverable
The gap doesn't arise from weak chips or careless teams. It comes from how the work is packaged. A typical AI workload, one "whole piece" of AI work, runs in stages, and only one stage needs the expensive chips. The standard way to run it holds that chip for the entire work run, including long stretches where the work never touches it. Pay for the chip the whole time, use it for a sliver.
Because the gap (let's call it what it is: waste) comes from how the work is packaged, the fix sits in the same place: change how the work is run. Send each stage of work to the chip that fits it best, hold the expensive chip only for the parts of the work that need it. The cost clock and the work clock move back toward each other. It happens on the hardware the business is already paying for, which is exactly why it's the recoverable budget line rather than a permanent tax.
Three questions make the gap actionable
No one needs to run their systems to hold their spend to account. Three plain questions do it, each targeting the gap between the two clocks:
- What share of the chips we pay for is doing useful work, measured over a normal week including nights and weekends? This is the work clock against the paid-for clock. A straight answer is one number over a real stretch; a weak answer is a peak figure or a number with no time window.
- Of this quarter's compute spend, how much bought finished work and how much bought idle time? This frames the gap in dollars. If most of it bought idle time, that's sunk cost waiting to be recovered, not the cost of doing AI.
- Can we recover that idle time on the hardware we already have, without a new contract, a longer commitment, a provider migration, or more hardware? This separates the cheapest cost lever - recovery of money already committed - from the expensive ones that require spending more money, and taking more time.
A note on the numbers
The per-hour & purchase prices here appear as ranges current to Q2 2026. There's no single market price for these chips: the same hardware rents for 2-5x more on the big clouds than on specialist providers, and that spread is carried here rather than collapsing to one, arbitrary figure. The 5% usage figure is from direct measurement across tens of thousands of production systems and comes with what it measures: the share of paid-for capacity doing useful work across ordinary enterprise fleets. The twenty-to-one framing is simply that figure restated, not a separate claim. The value-loss point for owned chips reflects an active accounting debate over how fast this hardware should be written down, and stands here as disputed rather than settled. The points made herein don't rest on any single number, but the shape of all of them together: a meter tied to time held, a return tied to work done, and a gap between them that is money already spent.
References
¹: H100 rental at roughly 2.00 to 3.50 dollars per GPU-hour on specialized providers, with a market median near 2.29 to 3.12 dollars across more than 40 providers, and specialized clouds 50 to 75 percent cheaper than hyperscalers for the same hardware. CloudZero; getdeploying.
²: The same H100 on-demand at about 6.88 dollars per hour on AWS and about 12.29 dollars per hour on Azure, 2 to 5 times the specialist-provider rate, billed by the hour while reserved. Spheron GPU cloud pricing comparison (May 2026).
³: NVIDIA H100 at roughly 25,000 to 30,000 dollars (PCIe) and 35,000 to 40,000 dollars (SXM5); a full eight-GPU system exceeding 350,000 dollars; hardware that loses value on a calendar as new architectures ship every twelve to eighteen months. CloudZero (May 2026); National Law Review on useful-life debate (Dec 2025).
⁴: Average GPU utilization of about 5 percent measured directly across tens of thousands of production clusters, indicating roughly 20 times over-allocation. Cast AI, 2026 State of Kubernetes Optimization Report.
⁵: Independent reporting on the same finding: about 5 percent average use across roughly 23,000 clusters. ITBrief.
The meter runs on time held; the value comes back on work done. The gap is money already spent, and it's the one cost you can recover without buying more of anything.