Inference-Aware Routing for LLMs: What Datadog Now Monitors

Standard Kubernetes load balancing routes traffic to the least-loaded backend pod. For LLM inference workloads, that definition of “least loaded” produces the wrong result: a pod with available CPU that doesn’t hold the relevant KV cache for a follow-up request will recompute the entire context from scratch, inflating time-to-first-token for the user and wasting GPU cycles on work that a different pod has already done.

The Kubernetes Inference Extension addresses this by routing requests based on backend serving state rather than generic resource metrics. Datadog has added monitoring capabilities that surface whether that routing is working as intended — a distinction that matters because routing misconfiguration and genuine capacity saturation produce similar observable symptoms until you disaggregate the signals.

What the Inference Extension Does

The Kubernetes Gateway API’s Inference Extension operates through three interconnected components:

Gateway routing validates incoming requests and extracts the target model from paths, headers, or JSON payloads using a Body-Based Router. It identifies which InferencePool the request should enter based on the declared model.

Endpoint selection consults an Endpoint Picker (EPP) that evaluates pod-level telemetry before selecting a destination. The EPP prioritizes pods that already hold relevant KV cache entries (reducing recomputation), pods with the required LoRA adapter already loaded (eliminating cold-start penalties), and pods with lower queue depth. Requests that exceed payload size thresholds are rejected with 413 before ever reaching a backend.

Flow control buffers requests in a central queue when all backends are saturated, dispatching based on priority tiers configured in InferenceObjective resources. Under sustained saturation, the system sheds low-priority requests with 429 or 503 responses rather than degrading uniformly across all traffic. This preserves interactive workload quality by deprioritizing batch jobs.

The full routing strategy details are documented by Datadog, including the four routing modes: model-aware, LoRA adapter-aware, prefix-aware (for KV cache locality), and priority-based shedding.

The Four Monitoring Layers

Datadog’s integration with inference workloads operates at four levels, each providing visibility into a different failure mode:

vLLM metrics track the backend serving layer directly. vllm.time_to_first_token.seconds is the primary outcome metric — increases here are what users experience. vllm.gpu_cache_usage_perc identifies cache pressure, and vllm.num_requests.running versus vllm.num_requests.waiting distinguishes active processing from queue buildup.

OpenMetrics from the Inference Extension expose routing-layer signals. inference_pool_per_pod_queue_size shows traffic distribution across backends — uneven distribution indicates EPP is applying policies correctly (or incorrectly). inference_extension_flow_control_request_queue_duration_seconds surfaces shedding events, which would be invisible to monitoring that only observes the backends themselves.

GPU monitoring provides hardware context via nvml.gpu_utilization, thermal states, and memory pressure. This layer is critical for distinguishing routing inefficiency from physical capacity limits: both produce elevated TTFT, but only one is fixable through configuration changes.

Kubernetes monitoring tracks pod restarts, OOMKills, and node status. A pod restarting under inference load is a different problem than a pod receiving poor routing decisions from the EPP.

Why Disaggregation Matters

The monitoring architecture matters because the visible symptom — degraded time-to-first-token — has multiple root causes that require different responses. Elevated TTFT from EPP sending requests to pods without the relevant KV cache is fixed through cache locality configuration. Elevated TTFT from genuine cluster-wide GPU saturation is fixed through capacity addition. Elevated TTFT from misconfigured flow control shedding interactive requests in favor of batch work is fixed through InferenceObjective priority tuning.

Without visibility across routing, serving, and infrastructure layers simultaneously, platform teams are effectively guessing which intervention to apply. The LangGraph agent monitoring pattern that Datadog has supported applies the same principle to agent-level observability — inference workloads and agent workflows share the same fundamental challenge of needing correlated visibility across multiple operational layers.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.