OpenCost 1.121.0: Now You Can See What Your Self-Hosted AI Models Actually Cost Per Token

Platform teams running AI inference on Kubernetes have been able to track GPU spend and token throughput separately. OpenCost 1.121.0 connects them — and the result changes the math on self-hosting vs. external APIs for models running at low utilization.

What Happened

The CNCF OpenCost project released version 1.121.0, adding the first open-source Kubernetes-native inference cost tracking capability. The release integrates with vLLM and llm-d (a CNCF sandbox project for distributed LLM inference) to generate per-model and per-token cost metrics as native Prometheus outputs. Proof-of-concept validation ran across “a cluster with 109 GPUs and 30 deployed AI models,” according to the CNCF blog.

How the Math Works

OpenCost 1.121.0 introduces two distinct cost metrics that platforms have previously had to calculate manually or approximate:

Allocation-based cost covers all infrastructure tied to keeping a model available — GPU memory reservation for model weights, active inference compute, and shared infrastructure overhead. This is the total cost of maintaining the model in a ready state.

Usage-based cost captures only the compute consumed during active inference, accounting for KV cache hit savings. This isolates the actual marginal cost of processing a token request when the model is already loaded.

The critical insight this combination enables: a model running at 25% GPU utilization that costs $1 per million tokens in raw compute actually costs $4 per million tokens when allocation overhead is included. At that utilization level, an external API at $2 per million tokens is the more economical choice — but without this measurement, most platform teams were comparing against the compute cost only and concluding self-hosting was cheaper.

The metrics are published as standard Prometheus outputs, meaning they integrate into existing Grafana dashboards and alerting workflows without additional tooling.

What It Signals

This release formalizes a calculation that previously required manual analysis or custom tooling. For platform engineering teams managing self-hosted model inference, three decisions now have a concrete cost basis: which models to self-host vs. route to external APIs, what utilization threshold makes self-hosting cost-effective, and how to allocate GPU resources across a multi-model cluster to maximize cost efficiency.

The integration with llm-d is notable because it’s a distributed inference runtime — the cost visibility extends to deployments that shard large models across multiple nodes, not just single-GPU inference endpoints.

For platform teams already using OpenCost for Kubernetes resource cost visibility, the broader platform engineering observability stack provides context for where inference cost tracking fits in the full FinOps picture.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.