Cloud Native Is Now AI-Native: Three Kubernetes Primitives That Don't Work for AI

Kubernetes was designed for microservices: small, stateless, independently scalable units. AI workloads are the opposite — massive, tightly coupled, initialization-heavy processes that behave like enormous monoliths. The cloud native infrastructure built over the last decade doesn’t fit them, and a KubeCon roundtable featuring engineers from AWS, Google, and Microsoft just laid out exactly where the gaps are.

How It Works

The CNCF roundtable identified three Kubernetes primitives that fail under AI workloads:

Pod scheduling without coordination: Standard Kubernetes schedules pods independently. AI training jobs require tightly coupled pod groups where all workers must initialize simultaneously before any begin computation. If one pod is delayed or fails, the entire job hangs. The solution being built is the Pod Groups API (Workload API), which treats a set of pods as a single failure domain — scheduling them as a unit rather than independently.

Static resource allocation for GPUs: The standard Kubernetes resource model allocates fixed CPU and memory at pod creation. AI workloads need dynamic access to GPUs, TPUs, and specialized accelerators that appear and disappear based on job requirements. Dynamic Resource Allocation (DRA) extends the Kubernetes scheduler to handle these resources as first-class citizens — requested at job creation, provisioned from a pool, and returned when the job completes.

Generic network routing for model I/O: Gateway API was designed for standard HTTP traffic. AI inference creates a new traffic class: prompt requests with variable-length responses, streaming outputs, model routing decisions, and guardrail enforcement. Inference Gateways adapt the Gateway API standard specifically for this prompt-in / token-out pattern, applying routing logic that standard gateways can’t handle.

The roundtable also flagged a practice shift in how AI changes the engineering process itself: product managers are now generating AI-driven prototypes before formal requirements documents. The volume of AI-generated code is creating review bottlenecks that teams are only beginning to structure around.

What This Means for You

Platform teams running Kubernetes-based infrastructure should treat these three gaps as current debt, not future planning. Pod Groups, DRA, and Inference Gateways are actively in development across the CNCF ecosystem — which means the Kubernetes clusters that ML teams are deploying on today will need upgrade paths as these APIs mature.

For teams already running AI workloads on Kubernetes, the practical short-term action is auditing which gaps are creating the most operational friction right now: scheduling failures, GPU resource contention, or inference routing complexity. Each maps to a specific CNCF development track with a known timeline.

The roundtable also noted the emerging role of agentic SRE — AI agents assisting with root-cause analysis while humans retain decision authority over critical changes. For platform teams, this suggests that observability tooling and incident response runbooks will need to be structured for agent-assisted analysis, not just human readers.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.