The headline number measures the wrong thing
Most teams gauge GPU use from one source: the utilization figure in nvidia-smi, or its higher-resolution twin in DCGM. That number is real, easy to read, and does not mean what most people assume.
The definition is the problem. GPU utilization is the percent of time over a sample period during which one or more kernels were executing, with the sample period set between one-sixth of a second and one second.¹² It measures whether anything ran, not how much of the chip ran or how fast. A single small kernel that occupies a sliver of the GPU for the length of the window scores 100 percent for that window.¹ The counter says fully busy. The hardware is mostly idle.
The gap surprises people the first time they see it. A training workload can show 100 percent GPU utilization while delivering 20 percent of the GPU's theoretical compute.³ A vLLM inference server can read 97 percent across an eight-minute window while token throughput drops threefold inside that same window.⁴ The dashboard is green, the work is not getting done, and the counter has told nobody anything useful.⁴ If the instrument reports busy when the truth is idle, every decision downstream of it is made blind.
Three different things called utilization
One word covers three different measurements, and people quote whichever is highest without saying which. Separating them is the first step.
Fleet utilization is the aggregate: across all the GPUs owned, what fraction is doing useful work on average. This is the number near 5 percent across tens of thousands of production clusters, and it captures the largest waste, accelerators allocated to something but running nothing.⁵ It answers the capital question: how much of the hardware bought is earning its place.
Server-capacity utilization is the per-device counter: on this GPU, right now, is a kernel running. This is what nvidia-smi and the default DCGM field report, and it is the one most prone to reading high while the chip is underused, because it measures occupancy of the sample window, not of the silicon.¹²
Model-FLOPs utilization, MFU, is the efficiency number: the ratio of the compute a workload delivered to the theoretical peak the hardware could deliver. It comes from Google's PaLM work and it exposes the gap, because a well-tuned training run lands at 35 to 45 percent MFU even when the busy-counter reads near 100.³⁶ It answers the real question, how much of the machine's capability turned into work, and it is the one almost nobody monitors continuously, because it is parameter- and framework-dependent and harder to compute than reading a counter.³⁶
These are not interchangeable. A team that conflates them quotes a high server-capacity number, believes its fleet is efficient, and never looks at the MFU that would say otherwise.
Why the counter lies even when it is precise
The reflex fix is finer-grained counters, and DCGM provides them: SM activity, tensor-core activity, memory-copy utilization, and on MIG the higher-precision graphics-engine-active field.⁷⁸ They help. They are still counters, and a counter measures presence, not productivity.⁴
A kernel running at 5 percent of peak FLOPS for the whole window still scores 100 percent on SM-active for that window.⁴ A GPU can show 90 percent SM utilization while its tensor cores sit idle, which on an AI workload means the most valuable hardware on the chip is doing nothing while the dashboard looks healthy.⁹ High memory-controller activity with low core activity is the signature of a job stalled waiting on data, busy by the counter, blocked in fact.¹⁰ Each finer counter narrows the lie. None closes it, because the only metric that captures useful-work-over-possible-work is the efficiency ratio, and that is the one the counters are not. Power draw is a cruder but more honest proxy, harder to fake: a chip delivering 5 percent of peak does not draw the power of one delivering 90.¹⁰
How to instrument it for real
The fix is a discipline, not a tool: never quote one number, and never quote the busy-counter alone.
Track multiple signals together. Core activity shows whether the SMs are busy, memory activity whether the job is data-stalled, tensor activity whether the AI-specific hardware is engaged, and power draw sanity-checks all three.⁹¹⁰ A reading is trustworthy only when the signals agree.
Deploy fleet-wide. DCGM exporter as a DaemonSet across every GPU node, scraped into a time-series backend, is the standard production pattern, and it turns per-device readings into a fleet number worth acting on.⁷¹¹ Single-node tools are for debugging one host; the capital question is answered only at the fleet level.
Compute MFU where it matters. You cannot watch MFU on every layer of every job, but you can compute it on the workloads that dominate spend, and that single number says more about whether a recovery opportunity exists than any dashboard of counters. Cross-stack profilers now instrument CPU stacks, GPU kernels, and collective-communication waits together at sub-half-percent overhead, which makes the measurement cheaper than it used to be.¹²
Span the whole pipeline, not just the accelerator. A job that is mostly CPU work holding a GPU reads idle on the GPU and explains nothing. Measuring the GPU alone hides the mismatch that created the waste.
The soft qualifier
This doubles as a way to know whether there is a problem worth solving. Four questions:
- Is the fleet utilization known, the real one, not the busy-counter average?
- Is the MFU of the top three workloads by spend known?
- When the dashboard reads high, is it known whether the tensor cores were engaged or idle?
- Can CPU and GPU activity for the same job be seen side by side, so it is clear when an accelerator is held by work that does not need it?
A team that can answer all four is instrumented to recover capacity and will know how much is there. A team that cannot is likely sitting on a large gap it cannot see, reassured by a green number that measures the wrong thing. The measurement is not a hurdle someone else imposes. It is the first piece of the recovery, and it pays for itself the moment it shows the distance between what was bought and what is used.
The point
You cannot fix what you cannot see, and the standard utilization metric does not let you see it. It reports whether a GPU is busy, not whether it is doing useful work, and those diverge enough that a fleet can read near full and deliver a fraction. Fleet utilization, server-capacity utilization, and model-FLOPs utilization are three different numbers, and quoting the easy one as if it were the honest one is how the gap stays hidden. Measure all of them, across the whole pipeline, and the recovery opportunity stops being invisible. The number on the dashboard is not the number you think it is.
A note on the numbers
The figures here are reported with their source and what they measure, which is the point of the piece. The three utilization definitions, fleet, server-capacity, and model-FLOPs, are kept strictly separate, because conflating them is the failure described. The 5 percent figure is fleet-level average GPU utilization across tens of thousands of production clusters; the 35 to 45 percent figure is MFU on well-tuned training; the near-100 percent readings are the busy-counter on specific workloads where measured throughput was far lower, cited to show the divergence, not as efficiency claims. The counter definition, one or more kernels executing over a sample window of one-sixth to one second, is from NVIDIA's own documentation. As across the series, the argument does not rest on any single number. It rests on the shape of all of them together: a headline metric that measures presence while the metric that measures productivity goes unwatched.
References
- nvidia-smi GPU utilization is defined as whether one or more kernels are executing over a sample period of one-sixth to one second; a small kernel that does not fill the window still scores 100 percent for that window. RL-Scope (arXiv 2102.04285), citing NVIDIA's documentation. arXiv
- DCGM exposes the same utilization number at finer granularity (DCGM_FI_DEV_GPU_UTIL) plus per-engine counters; the underlying definition remains time-with-a-kernel-running, not work delivered. Datadog NVML documentation as the clearer published definition; DigitalOcean real-time monitoring guide (Apr 2026). DigitalOcean
- A training workload reading 100 percent GPU utilization while delivering roughly 20 percent of theoretical compute; MFU introduced in Google's PaLM work as the ratio of observed to theoretical-peak throughput, parameter- and framework-dependent and hard to monitor continuously. Trainy "GPU Utilization is a Misleading Metric"; MarkTechPost on SM efficiency. Trainy MarkTechPost
- A vLLM server reading 97 percent GPU utilization while token throughput drops threefold in the same window; per-engine counters (SM_ACTIVE, TENSOR_ACTIVE, MEM_COPY_UTIL) remain counters, and a kernel at 5 percent of peak FLOPS still scores 100 percent SM_ACTIVE. Ingero, "GPU Utilization Is a Counter, Not a Cause" (May 2026). Ingero
- Average GPU utilization of 5 percent across tens of thousands of production clusters; the largest waste is capacity allocated but running nothing. Cast AI 2026 State of Kubernetes Optimization Report; ITBrief reporting. Cast AI ITBrief
- Most LLM training runs land at 35 to 45 percent MFU; SM efficiency recommended as a more representative signal than GPU utilization, with MFU more accurate still but harder to compute layer by layer. Trainy; CoreWeave H100 benchmarks corroborate the 35 to 45 percent band. Trainy CoreWeave
- DCGM exporter deployed as a DaemonSet exposing per-device metrics (including DCGM_FI_PROF_PIPE_TENSOR_ACTIVE for tensor cores) to a Prometheus backend; true utilization requires reading compute, memory, tensor, and power together rather than one metric alone. DevZero complete guide to measuring GPU utilization. DevZero
- On MIG, DCGM_FI_PROF_GR_ENGINE_ACTIVE gives higher-precision graphics-engine-active measurement, with overall utilization computed as a slice-weighted sum across instances. WhaTap on MIG utilization monitoring. WhaTap
- A GPU can show 90 percent SM utilization while tensor cores sit idle, meaning an AI workload is not using the chip's most valuable hardware. DevZero. DevZero
- High memory-controller activity with low core activity is the signature of an input-stalled job; tracking core, memory, temperature, and power together rather than one signal in isolation; power draw as a proxy for real computational work. DigitalOcean real-time monitoring guide. DigitalOcean
- DCGM exporter plus a monitoring backend as the standard pattern for fleet-level visibility, versus single-node CLI tools (nvidia-smi, nvtop) for per-host debugging. DigitalOcean. DigitalOcean
- Cross-stack profiling instrumenting CPU stacks, GPU kernel traces, and NCCL collective-communication events together via eBPF at sustained sub-0.4 percent overhead. SysOM-AI (arXiv 2603.29235), as summarized in the Ingero analysis. Ingero