GPU Autoscaling on Kubernetes with KEDA: Why CPU Metrics Fail AI Inference Workloads

Kubernetes’ default Horizontal Pod Autoscaler reads CPU and memory — neither of which accurately reflects whether a GPU-accelerated inference pod is actually under load. A pod running an LLM can sit at 5% CPU while consuming 80% VRAM and processing requests at capacity. The CPU-based HPA sees idle capacity and either fails to scale up or scales down prematurely, wasting expensive GPU resources or degrading inference latency.

The Mismatch Between CPU Metrics and GPU Workload Reality

KEDA’s new GPU external scaler addresses this by running a DaemonSet on GPU nodes that reads hardware metrics directly from NVIDIA’s Management Library (NVML) via go-nvml. Those metrics are then exposed over gRPC using KEDA’s ExternalScaler interface, giving the KEDA operator real GPU signal to make scaling decisions.

The five metrics the scaler exposes are: gpu_utilization (compute percentage), memory_utilization (memory controller load), memory_used_percent (VRAM usage), temperature, and power_draw. For multi-GPU nodes, users choose an aggregation method — max, min, avg, or sum — or target a specific GPU index. This architecture mirrors Kubernetes device plugins, where a per-node agent collects hardware-specific data that the scheduler layer cannot see directly.

The EKS Auto Mode release earlier this year addressed node-level GPU provisioning. KEDA’s external scaler completes the picture at the workload level — once the nodes are provisioned, the pods need their own autoscaling logic that reflects actual GPU utilization rather than proxying through CPU.

Three Pre-Built Profiles and Scale-to-Zero

The implementation ships three pre-built workload profiles that provide sensible default targets:

The vllm-inference profile enables scale-to-zero: when VRAM drops below threshold, KEDA scales the deployment to zero replicas, freeing GPU resources entirely. For inference workloads with variable demand — batch processing, off-peak LLM endpoints, development environments — this is a meaningful cost lever. A GPU that scales to zero between jobs stops burning $2-8/hour in cloud GPU costs for idle capacity.

The CNCF’s AI production infrastructure report from earlier this year identified GPU waste as one of the top cost drivers for teams running AI workloads on Kubernetes. The KEDA external scaler directly addresses that finding with a mechanism that platforms can deploy against existing GPU node pools without infrastructure changes.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.