GPU Cost Attribution on Kubernetes: Bill on Memory, Not Compute

Put three pods on one shared GPU, and the standard exporter will report each of them using the whole card: 300% of a GPU that doesn’t exist. That example, from a New Stack analysis of GPU visibility on Kubernetes, sums up the problem for platform teams asked to charge GPU spend back to the teams using it. The most expensive line on the cloud bill is also the one Kubernetes tells you least about.

How It Works

With CPU and memory, the kernel tracks usage per container, so per-pod numbers come almost free. GPUs don’t work that way. NVIDIA’s DCGM exporter, the tool most teams start with, reports at the physical device level by default. That is accurate when a pod has a GPU to itself. Once pods share a GPU through time-slicing or MPS, the article explains, the driver exposes one set of counters for the whole card, so every pod gets the same value. The dashboards still look fine, which is what makes the error hard to catch.

Several other CPU assumptions also break:

• Requests: a GPU is a whole-number resource (nvidia.com/gpu: 1), requests and limits must match, and the scheduler doesn’t overcommit. Fractional sharing happens underneath, so the requested-versus-used comparison behind rightsizing barely exists.
• Guardrails: base CUDA images set NVIDIA_VISIBLE_DEVICES to all. If nothing overrides it, a pod can see every GPU on the node, whatever the scheduler recorded.
• Utilization: the usual metric says a kernel was running during the sample window, not how much compute or memory bandwidth was used. A GPU can read 100% while doing little, and frameworks like vLLM claim most of the memory at startup.
• Sharing modes: time-slicing, MPS, MIG, and Dynamic Resource Allocation each change what “allocated” means. The article notes that most clouds still charge for the whole GPU when you use a MIG slice, and A100 and H100-class cards split into at most seven instances.

Attribution is where the article draws its clearest line. NVML reports per-process GPU memory reliably, and each process can be traced to a pod and team. The totals fall slightly short of the card because the driver reserves some memory. Compute is different. The available signal is closer to a flag that records whether a job was doing anything at the moment it was checked. When two jobs genuinely run side by side, both read busy nearly all the time, and the article says nothing available today can tell you which of two teams sharing a GPU burned 70% and which burned 30%.

What This Means for You

The money makes this urgent. The article puts the cheapest MIG-compatible A100 instance on AWS at $22 an hour, about $14 on spot, against roughly $5 for a comparable CPU-only instance. The cloud bills a GPU node as one line item, so a per-workload number means splitting CPU, memory, and GPU apart, weighting by GPU model, and then dividing among the pods sharing each card. The hidden cost, the author argues, is the engineering time spent rebuilding visibility that CPU workloads get for free. Teams already chasing idle GPUs on Kubernetes will recognize the pattern.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.