Seven of Eight Agentic AI Pipeline Stages Run on CPUs — and Supply Is Running Out
Agentic AIThe GPU buildout narrative has dominated AI infrastructure discussions for three years, but agentic AI workloads are revealing a different constraint. According to IEEE Spectrum’s analysis, seven of eight stages in a realistic agentic pipeline run on CPUs — and server CPU supply is already running out.
What Happened
The GPU supply chain scaled to handle LLM inference, which is where the heavy matrix multiplication lives. But inference is only one stage in an agentic workflow. The surrounding orchestration — parsing model outputs, invoking tools, making API calls, executing code, collecting results, routing decisions — lands on CPUs.
AMD’s VP of compute quantified this directly: seven of the eight stages in a typical agentic pipeline are CPU-bound. The one GPU stage (language model inference) gets all the infrastructure attention; the other seven don’t.
Tokenization compounds the problem. In a multi-step agentic workflow, the model’s growing context window — often reaching 500,000 tokens or more across a long session — must be re-tokenized at every tool call step. That tokenization work runs sequentially on CPUs. As context lengths grow, the tokenization overhead multiplies faster than inference time, flipping the bottleneck.
The hardware consequence is already visible: Intel sold out server CPUs through year-end 2026. AMD doubled its server CPU forecast for the year. The shortage is trickling into consumer markets as manufacturers prioritize server chips for agentic infrastructure demand.
Why It Matters
This creates a specific planning problem for teams building agentic systems. Infrastructure sized for GPU-heavy inference workloads can have adequate GPU capacity but insufficient CPU cores to handle the surrounding orchestration at production scale. The GPU/CPU idling pattern — CPUs waiting during inference, GPUs waiting during tool execution — means both resources underutilize when the ratio is wrong.
Research shows that increasing CPU cores reduces time-to-first-token latency by 1.5 to 7 times depending on workload. For systems where user experience depends on agentic response speed, CPU allocation is a direct lever on perceived performance — one that was invisible when the dominant workload was single-shot chat inference.
This is a materially different problem from what Meta’s multibillion-dollar AWS Graviton5 deal was optimizing for in early 2026. The Graviton5 story was about replacing GPU inference for specific models; the agentic CPU bottleneck is about the orchestration fabric that surrounds every model call.
The So What
Infrastructure planning for agentic AI needs a CPU-first audit alongside the standard GPU sizing exercise. The question isn’t just “how many GPU cores for inference” but “how many CPU cores for the seven pipeline stages that surround each inference call.” Teams running agentic workloads at scale who haven’t done this audit are likely hitting CPU ceilings they’ve attributed to other causes. Given the current server CPU supply situation, procurement lead times are also longer than they were six months ago — another reason not to wait for the bottleneck to become a production incident before addressing it.
Source: IEEE Spectrum
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →