Kubeflow + Cilium: Why Your GPUs Sit 60% Idle

Sixty percent idle. Not because the GPUs weren’t scheduled — every pod was running, healthy, no crashes, no OOM kills. The training job was live and burning cloud spend the whole time it sat there. The problem wasn’t capacity. It was that two systems, each behaving exactly as designed, were quietly working against each other.

How It Works

Two Adobe engineers, Ramkumar Nagaraj and Bingi Narasimha Karthik — both Golden Kubestronauts — traced the pattern in a CNCF blog post walking through the bug they found in production. Their distributed training job showed GPUs pinned at roughly 40% utilization instead of the near-full saturation you’d expect from healthy pods and correct scheduling.

The root cause sat at the seam between two independently correct systems. Kubernetes’ scheduler places pods based on available CPU, memory, and GPU — it doesn’t know or care which availability zone a pod lands in. Cilium’s network policies, meanwhile, enforce topology-aware zone boundaries for security, compliance, and cost control — and once a pod is placed, Cilium can’t reschedule it.

Put those two together on a Kubeflow distributed training job and you get a coordinator pod landing in one zone while the GPU worker pods land in another. CiliumNetworkPolicy then silently interferes with the cross-zone connection those pods need for gradient synchronization.

The engineers found this played out three different ways, each with a different time-to-discovery:

• A hard zone-boundary block denies the connection outright and training never starts — obvious within seconds.
• Without a hard block, gradient sync pays a round-trip latency penalty every step, and throughput quietly drops 30 to 60% — the kind of degradation you might not notice for hours.
• Traffic crossing availability zones shows up nowhere except the cloud bill — an egress-cost surprise that takes days to surface, usually in an invoice.

What actually exposed the bug was GPU utilization metrics in Prometheus and Grafana, cross-referenced against pod-zone placement. With that instrumentation in place, the mismatch was visible in seconds. Without it, the same problem can hide for days behind a fleet of pods that all report healthy.

What This Means for You

If you’re running distributed GPU training on Kubernetes with a CNI that enforces topology-aware network policy — Cilium or otherwise — this bug class is worth checking for even when nothing looks broken. “All pods Running, zero crashes” is not the same as “the network path between them is fast.”

A few takeaways from the writeup:

• GPU utilization is a topology metric, not just a capacity metric. If your GPUs are underutilized despite correct scheduling, check where the pods actually landed relative to each other before you assume you need more hardware.
• The fix didn’t require touching Cilium or Kubeflow. The authors resolved it entirely with Kubernetes-native scheduling primitives: nodeAffinity to pin the training group to the GPU zone, topologySpreadConstraints to co-locate coordinator and workers, and a toleration to let the coordinator land on GPU-tainted nodes. For clusters that need a more portable answer, they note podAffinity keyed to zone topology as an alternative.
• Security and scheduling tooling can each be correct and still collectively wrong. Neither the Kubernetes scheduler nor Cilium’s policy engine was misconfigured. The failure only existed in the gap between them — exactly the kind of failure that doesn’t show up in either system’s own health checks.

After applying the co-location fixes, the authors report GPU utilization went from roughly 40% to roughly 85%. The team also published a reproduction lab on GitHub alongside a KubeCon India 2026 talk walking through the full debugging process.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.