Monitor LLM Routing with the Kubernetes Inference Extension and Datadog

• KIE routes LLM inference traffic across four strategies: model-aware, LoRA adapter-aware, prefix-aware, and priority-based with request shedding
• Agent spans (LLM decision-making) run 2× slower than tool spans — the model is the bottleneck, not the external APIs
• Scale-to-zero is viable for batch LLM workloads: GPU backends can scale to zero during idle, accepting minutes-long cold starts in exchange for eliminating idle GPU costs
• Datadog integrates via vLLM metrics + OpenMetrics with five environment variables of setup

Purpose

Instrument a Kubernetes Inference Extension (KIE) deployment with Datadog to observe LLM routing decisions, GPU backend utilization, and the four routing strategy performance characteristics. Enables cost optimization through data-driven GPU fleet sizing and scale-to-zero decisions.

Pre-requisites

• Kubernetes cluster with KIE gateway deployed
• vLLM serving GPU model backends
• Datadog Agent with APM and LLM Observability enabled
• Environment variables: ML_APP_NAME, DD_API_KEY, DD_ENV, DD_VERSION, DD_HOSTNAME

Steps

1. Understand the three-phase request lifecycle

KIE routes requests through three phases: gateway routing → external processor endpoint selection → model serving. Datadog traces each phase as a distinct span, enabling isolation of where latency accumulates:

• Gateway: routing strategy evaluation and backend selection
• External processor: LoRA adapter loading, prefix cache lookup, queue prioritization
• Model serving: inference execution, token generation

2. Configure the four routing strategies

For RAG pipelines and templated agent prompts, prefix-aware routing delivers the highest efficiency gain — requests sharing a common system prompt or document prefix can reuse cached KV computation, reducing inference latency and GPU memory pressure.

3. Monitor the critical metrics

# Datadog monitor — KV cache pressure
kv_cache_hit_rate: alert when < 0.6 (60% hit rate threshold)
queue_depth_imbalance: alert when backend variance > 30%
shedding_events_per_minute: alert when > 0 in production tier
adapter_load_frequency: track per-adapter to identify hot adapters for pre-loading

4. Configure scale-to-zero for batch workloads

Scale-to-zero eliminates idle GPU costs for async, non-interactive workloads. Configure KEDA or HPA with a zero-replica floor:

minReplicas: 0
scaleDownStabilizationWindowSeconds: 300
metrics:
  - type: External
    external:
      metric:
        name: llm_queue_depth
      target:
        type: Value
        value: "1"

Accept the trade-off: cold starts take 2–5 minutes for large models loading GPU memory. Scale-to-zero is appropriate for overnight batch jobs, scheduled analysis tasks, and development environments — not interactive serving paths.

5. Alert on the bottleneck, not the symptom

Because LLM reasoning spans run ~2× slower than tool spans, latency alerts should target model inference time directly — not downstream API response time. A slow Tavily search response is rarely the root cause; a slow vLLM inference call almost always is.

The Trap

Setting minReplicas: 0 on interactive serving paths. Users hitting a cold-start GPU backend will wait 2–5 minutes for a response. Use scale-to-zero only for workloads where async queuing is acceptable; maintain a warm baseline replica count for any synchronous user-facing path.

Cited References

• Monitor LLM routing with the Kubernetes Inference Extension — Datadog
• Kubernetes Inference Extension documentation
• vLLM metrics reference
• KEDA for Kubernetes autoscaling
• LangGraph agent monitoring with Datadog

Source: Datadog Blog

Content created with AI assistance and reviewed for accuracy.