Kubernetes v1.36: How to Use Pod-Level Resource Managers for NUMA-Aligned Workloads

Summary

• Kubernetes v1.36 (alpha) adds .spec.resources at pod level — a budget covering all containers
• Main containers can claim exclusive, NUMA-aligned CPUs from that budget
• Remaining resources form a “pod shared pool” for sidecars and init containers
• Eliminates the old tradeoff: no longer need to assign exclusive CPUs to every container to get Guaranteed QoS for any container
• Requires two feature gates and Topology Manager in pod or container scope

Purpose

Performance-critical workloads (databases, ML inference, latency-sensitive services) benefit from NUMA-aligned, exclusive CPU allocation — it eliminates cross-NUMA memory latency and prevents CPU throttling. Before v1.36, getting this required setting every container in the pod to Guaranteed QoS with integer CPU limits, including lightweight sidecars like log shippers and service mesh proxies. That forced teams to over-allocate exclusive cores to containers that had no use for them, or abandon the performance optimization entirely.

Pod-level resource managers introduce a pod-scoped budget. The main container claims exclusive cores from that budget; remaining resources form a shared pool that sidecars use without consuming dedicated cores.

Pre-requisites

• Kubernetes v1.36 cluster
• kubelet Topology Manager configured with pod or container scope (not none)
• Both feature gates enabled on the kubelet

Steps

1. Enable the feature gates on each kubelet node

Add to /etc/kubernetes/kubelet-config.yaml:

featureGates:
  PodLevelResources: true
  PodLevelResourceManagers: true

Restart the kubelet after applying the change.

2. Verify Topology Manager scope

Confirm Topology Manager is active:

kubectl get node <node-name> -o jsonpath='{.status.allocatable}'

Check the kubelet config for topologyManagerScope: pod or topologyManagerScope: container.

3. Define pod-level resources in the Pod spec

Set .spec.resources with the total budget for the pod. Then set individual container requests/limits from within that budget for containers requiring exclusive allocation:

apiVersion: v1
kind: Pod
metadata:
  name: db-with-sidecars
spec:
  resources:
    requests:
      cpu: "8"
      memory: "16Gi"
    limits:
      cpu: "8"
      memory: "16Gi"
  initContainers:
  - name: log-forwarder
    image: fluent/fluent-bit:latest
    restartPolicy: Always
  - name: service-mesh
    image: istio/proxyv2:latest
    restartPolicy: Always
  containers:
  - name: database
    image: postgres:16
    resources:
      requests:
        cpu: "6"
        memory: "12Gi"
      limits:
        cpu: "6"
        memory: "12Gi"

In this example, the database container claims 6 exclusive NUMA-aligned CPUs and 12Gi from the pod budget. The remaining 2 CPUs and 4Gi form the pod shared pool used by the log forwarder and service mesh sidecar.

4. Confirm exclusive CPU assignment

After the pod is scheduled, inspect CPU pinning on the node:

# SSH to the node running the pod
cat /sys/fs/cgroup/cpuset/kubepods/pod<uid>/<container-id>/cpuset.cpus

The database container should show a specific, non-overlapping CPU set. Sidecar containers will show a broader shared set.

5. Validate CFS quota behavior

Exclusive containers have CPU CFS quota enforcement disabled at the container level — the kernel scheduler will not throttle them within their allocated set. Sidecar containers in the pod shared pool have CFS quotas enforced at pod level:

cat /sys/fs/cgroup/cpu/kubepods/pod<uid>/<sidecar-container-id>/cpu.cfs_quota_us

A value of -1 on the main container confirms no throttling. Positive values on sidecars confirm pod-level enforcement.

The Trap

Pod-level resource specs and container-level resource specs are additive, not hierarchical — the scheduler enforces the pod budget, but individual container limits must still fit within it. If you set the pod budget to 8 CPUs and container limits totaling 10 CPUs, the pod will fail to schedule with a PodQOSClass mismatch error that does not clearly surface the overcommit. Always verify that the sum of all container requests in the pod does not exceed .spec.resources.requests.

Cited References

• Kubernetes v1.36 Pod-Level Resource Managers announcement
• Kubernetes Topology Manager documentation
• CPU Manager for Kubernetes

Photo by Jan van der Wolf on Pexels

Source: Kubernetes Blog

Content created with AI assistance and reviewed for accuracy.