Why Queue Depth Beats CPU as a Kubernetes Autoscaling Signal

A worker pod can sit at 5% CPU while a queue behind it fills with thousands of unprocessed messages — and the Horizontal Pod Autoscaler, watching only CPU, will see nothing wrong.

How It Works

Kubernetes’ default autoscaler reads CPU and memory utilization, metrics that describe resource pressure but not actual work backlog. For event-driven workloads — anything consuming from a queue rather than serving synchronous requests — that’s the wrong signal entirely. A worker can be idle by every infrastructure metric while messages pile up in Amazon SQS, because polling a queue and waiting for the next message doesn’t spike CPU the way request handling does. Backlog, not infrastructure utilization, is the metric that actually reflects whether the system is falling behind.

KEDA (Kubernetes Event-Driven Autoscaling), a CNCF project, closes that gap by sitting between the event source and Kubernetes’ Horizontal Pod Autoscaler as a translator. It polls SQS directly and computes outstanding messages as ApproximateNumberOfMessages + ApproximateNumberOfMessagesNotVisible, then derives the target replica count as ceil(outstanding messages / queueLength). With a queueLength of 10 messages per pod, that formula produces 0 pods at 0 messages (true scale-to-zero), 3 pods at 25 messages, and 10 pods at 95 messages — capped by whatever maxReplicaCount the operator sets.

The implementation requires a handful of pieces beyond installing KEDA itself via Helm: a least-privilege IAM policy scoped to GetQueueAttributes, ReceiveMessage, and DeleteMessage; a TriggerAuthentication resource wired to AWS pod identity; and a ScaledObject resource that defines the scaling parameters. The tunables that matter most in practice are activationQueueLength (the threshold for scaling up from zero), cooldownPeriod (how long to wait before scaling back down — 120 seconds in the reference example), and pollingInterval (how often KEDA checks the queue — every 10 seconds in the same example).

What This Means for You

The practical payoff is real cost and latency improvement for any team running asynchronous, queue-driven services on Kubernetes: burst traffic gets handled faster because scaling responds to actual backlog instead of lagging resource metrics, and idle periods genuinely scale to zero instead of holding pods open on the assumption that “low CPU” means “nothing to do.” That’s the same fundamental gap KEDA’s GPU-based scaler addresses for AI inference workloads — CPU and memory are the wrong proxy whenever the real constraint lives somewhere else, whether that’s VRAM utilization or message backlog.

The pattern generalizes past SQS specifically — the same backlog-driven logic applies to any messaging platform with a queryable depth metric, Kafka topics or RabbitMQ queues included, which is why KEDA supports dozens of scalers beyond AWS. But tuning it correctly requires deliberate choices, not defaults left alone: queueLength should reflect your actual per-message processing time, not a guess, and cooldownPeriod needs to account for whether your workload’s traffic is genuinely bursty or just noisy. Get those wrong and you either scale too aggressively (cost) or too slowly (latency) — the formula does exactly what it’s told, so what it’s told matters. Operational visibility matters too: when scaling doesn’t behave as expected, the debugging path runs through KEDA’s own logs, HPA status, and the raw queue attributes, not just pod-level metrics.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.