Goodput, Not Throughput, Is the Right Metric for LLM Serving

Platform teams optimizing LLM infrastructure for throughput are measuring the wrong thing. A CNCF benchmarking study found two configurations with nearly identical throughput numbers — one delivered smooth responses; the other had p95 TPOT nearly 10x higher, producing responses so stuttered they were noticeably degraded for end users.

What Happened

The study introduces goodput as the operationally honest alternative to throughput for production LLM serving. Throughput counts raw requests processed per second. Goodput counts only requests that meet defined latency service level objectives — specifically Time to First Token (TTFT) and Time Per Output Token (TPOT). A system can improve throughput while simultaneously failing more users by missing those latency targets.

The restaurant analogy holds: throughput measures plates leaving the kitchen; goodput measures satisfied diners. A kitchen pushing out twice as many plates while half the orders arrive cold is not a better kitchen.

Three workload patterns were benchmarked against different latency constraints: chatbot workloads (TTFT under 1.5 seconds), reasoning workloads (no TTFT constraint, throughput ceiling), and agentic workloads (per-call TTFT accumulation). The agentic results were the sharpest: nearly tied throughput (6.20 vs. 6.11 requests/sec) but 220ms TPOT variance — enough to degrade multi-step agent task completion in ways that compound across calls.

Only three vLLM parameters were varied: gpu_memory_utilization, max_num_batched_tokens, and max_num_seqs. The solution space still contained “small cliffs” — configurations that hand-tuning reliably misses — requiring systematic automated search rather than manual optimization.

Why It Matters

“A throughput number with no SLO next to it is marketing, not engineering.” The study makes this point not as editorial commentary but as a practical finding: the configuration that won the throughput benchmark was not the configuration that served users well. If your team is reporting LLM serving performance without attached latency SLOs, you are optimizing a metric that can be gamed against user experience.

The agentic workload case is particularly relevant as more production systems move toward multi-step agent architectures. TTFT accumulates across tool calls. An inference stack with acceptable TTFT for a single-turn chatbot may degrade significantly when the same model is called twelve times in a sequential agent loop. The benchmark tested this directly and found the variance meaningful.

For teams running AI workloads on Kubernetes, the compounding effect of suboptimal LLM serving configuration is a silent cost: slower agents, more retries, higher infrastructure spend for equivalent user-perceived throughput. Continuous re-optimization is also required — what was the optimal configuration for one traffic pattern and model version drifts as prompts, usage patterns, and models evolve.

Source: CNCF

Content created with AI assistance and reviewed for accuracy.