TAHO Labs
All Insights

Industry Analysis

The Agentic CPU Tax

Photo of Todd Smith
Todd Smith · June 16, 2026 · 7 min read

What an agent does

A chatbot was simple: prompt in, GPU forward pass, tokens out. The CPU handled routing and tokenization, fast enough to never show up in profiling.¹

An agent is a loop. Plan, act, reflect, revise. It decides what to do, calls a tool, reads the result, decides whether it is done, and either fires the next step or starts over. The loop is sequential and branchy, and almost all of it runs on the CPU: tool calls, API requests, database queries, output parsing, state management, and the orchestration that strings sub-agents together.¹² The GPU does the inference. The CPU does everything between inference calls, and there is a lot of between.

The numbers are stark. A Georgia Tech and Intel study put tool processing on CPUs at 50 to 90 percent of total latency in agentic workloads.³ When a tool call averages 500 milliseconds and inference averages 200, the agent spends roughly 70 percent of its time waiting on CPU work.² During all of it, the GPU is idle, holding its allocation, drawing its cost, doing nothing.

This is the shape change, sharpened

The series has a name for the gap between what work needs and what the box provides. Agentic work does not introduce a new problem. It makes the existing one worse on purpose.

A standard inference workload already moves between CPU-heavy and GPU-heavy phases. An agent runs that switch dozens of times inside a single task, and weights it toward the CPU side. Reinforcement learning multiplies it again: rollouts, environment steps, and reward evaluation are CPU and I/O bound work wrapped around the GPU forward pass, so an RL job can be mostly CPU with GPU bursts.⁴ The heterogeneity that the shape-change argument described as a structural trend is, in agentic and RL workloads, the dominant feature of the work.

The hardware vendors have already conceded the point. NVIDIA co-packages a Grace CPU with its Blackwell GPU in the GH200 and GB200 precisely because agentic work needs more CPU alongside the accelerator.² Arm projects CPU core demand rising fourfold per gigawatt of AI data center in the agent era, from roughly 30 million cores per gigawatt to 120 million.⁵⁶ Intel has said Xeon demand exceeds supply by a meaningful amount, with server CPU prices up 27 percent in the first quarter of 2026 and the CPU-to-GPU ratio tightening from 1:8 toward 1:1 in agentic scenarios.⁶ The whole industry is rebalancing toward CPU because the work demanded it.

Why buying more GPUs makes it worse

Here is the trap. When an agent fleet shows low GPU utilization, the reflex is to read it as a capacity problem and add accelerators. The diagnosis is backward.

If the GPU sits at 30 to 40 percent while the CPU is pegged, the bottleneck is the CPU, and adding GPU capacity does nothing but add idle accelerators.² The agent is not starved for inference. It is waiting on tool calls. More GPUs widen the gap between what is bought and what is used, on the most expensive hardware in the building, for a workload that was already GPU-idle the majority of the time. The CPU tax is paid in idle GPU hours, and the standard response makes the bill larger.

The structural reason is the bundled unit. An agent is typically packaged as one container, the CPU loop and the GPU inference together, and placed on a GPU node. So the accelerator is held for the entire Plan-Act-Reflect-Revise cycle even though it is needed only for the inference bursts inside it. The pod holds the GPU through 50 to 90 percent of the latency that is pure CPU work.³ You cannot schedule your way out of that, because the waste is inside the unit. Placement decides which node the bundled agent lands on. It does not split the CPU loop from the GPU bursts.

The fix is the thesis, and it ages forward

The answer is the same one the series has argued throughout, and agentic work makes it more valuable, not less. Break the unit. Run the CPU loop, the planning, the tool calls, the parsing, the orchestration, on CPU capacity, and send only the inference bursts to an accelerator that stays fed because it is shared across many agents instead of held idle by one. The GPU stops waiting on tool calls because it is no longer chained to a single agent's loop.

This is a layer beneath the orchestrator and above the silicon, routing each unit of work to the resource that fits it, across heterogeneous hardware. Agentic and RL workloads are the case where that routing matters most, because they are the workloads where the CPU-GPU mismatch is widest and the idle-GPU cost is highest. And the trend runs one direction. Agents are the larger share of valuable AI work every quarter, inference is already roughly two-thirds of all AI compute, and the CPU side of the ledger is growing fourfold.⁵⁶ The mismatch this layer closes is not a fixed target. It is widening, which means the value of closing it compounds as the work moves where it is going.

The point

Agents spend most of their time on the CPU, and the GPU sits idle through the planning, the tool calls, and the retries. That is the shape change at its sharpest: the work is mostly not-GPU, and it is bundled onto GPU nodes anyway. Adding accelerators to a CPU-bound agent fleet buys more idle GPU. The fix is to break the unit and route the CPU loop and the GPU bursts to the resources that fit them, which is exactly where the execution layer sits. The gap is widening as agents take over, so the value of closing it grows with the work. The agentic era does not weaken the case for fit. It is the strongest case there is.

A note on the numbers

Figures are reported with their source and what they measure. The 50 to 90 percent CPU-latency figure is from a specific academic and vendor study on representative agentic workloads, and the 70 percent waiting figure is a worked example from stated tool-call and inference latencies, not a fleet average. The CPU-to-GPU ratio numbers, 1:8 tightening toward 1:1, and the fourfold core-per-gigawatt projection are industry and vendor estimates for the agent era and are labeled as projections. The two-thirds inference share and the CPU pricing and supply figures are current to Q1 and Q2 2026 and will move. The argument does not depend on the precision of any of them; it depends on the direction, which every source agrees on: agentic and RL work is more CPU-weighted than what came before, and the weighting is increasing. As across the series, the argument does not rest on any single number. It rests on the shape of all of them together.

References

  1. Traditional batch inference is almost entirely GPU-bound, with CPU routing and tokenization fast enough to never appear in profiling; agentic pipelines break that assumption because every tool call, environment step, tokenization pass, and orchestration decision runs on CPU. Spheron CPU-to-GPU ratio guide for agentic inference (Jun 2026). Spheron
  2. Between inference calls agents execute tool calls, database queries, output parsing, state management, and downstream orchestration on CPU, so the GPU can be idle longer than it is active; GPU at 30 to 40 percent with CPU pegged means a CPU bottleneck that adding GPU will not fix; a 500ms tool call against 200ms inference leaves the agent waiting roughly 70 percent of the time; NVIDIA co-packages Grace CPU with Blackwell GPU in GH200 and GB200 because agentic work needs more CPU alongside the accelerator. ParallelIQ on CPU vs GPU bottlenecks in agentic AI (May 2026). ParallelIQ
  3. A Georgia Tech and Intel study (Nov 2025) quantified tool processing on CPUs at 50 to 90 percent of total latency in agentic workloads, with the GPU idle while the CPU finishes before the next batch. UncoverAlpha on the CPU bottleneck of the agentic era; corroborated by the CPU-centric agentic-execution characterization. UncoverAlpha arXiv
  4. Reinforcement learning adds rollouts, environment steps, and reward evaluation as CPU and I/O bound work wrapped around the GPU forward pass, multiplying the CPU share of the workload. Viks Newsletter on the CPU bottleneck in agentic AI. Viks Newsletter
  5. Arm estimates AI data center CPU demand rising from roughly 30 million cores per gigawatt in the LLM era to 120 million in the agent era, a fourfold increase, with the future CPU-to-GPU ratio shifting toward 1:1 to 1:2. TrendForce on the agentic CPU-GPU rebalance (Apr 2026). TrendForce
  6. Inference roughly two-thirds of all AI compute in 2026 (up from a third in 2023); server CPU prices up 27 percent in Q1 2026 with Intel citing Xeon demand exceeding supply and the CPU-to-GPU ratio tightening from 1:8 toward 1:1 in agentic scenarios; Arm projecting fourfold CPU core growth per gigawatt. Nirvana Labs on why agentic AI needs more CPUs. Nirvana Labs