Why the controller-runtime Cache Keeps Kubernetes Stable

Every custom Kubernetes controller has to answer the same question thousands of times a day: what’s the current state of this resource? If it asked the API server directly every time, a cluster running even a modest number of controllers would grind the control plane to a halt. The reason that doesn’t happen is a piece of infrastructure most controller authors never think about — the controller-runtime cache.

How It Works

When you scaffold a controller with controller-runtime — the standard path via kubebuilder, and the plumbing under most operators you’ll ever write — calls like r.Get() and r.List() inside your reconcile loop don’t hit the Kubernetes API server at all. They read from a local, in-memory cache. That cache is built with a list-and-watch pattern borrowed directly from how kubectl get pods --watch works under the hood: the manager performs one initial list call to populate the cache, then opens a watch subscription that streams every subsequent change and keeps the cache synchronized in real time. Reads become essentially free, a local memory access instead of a network round trip, which is what lets a controller run hundreds of reconciliation loops per second without adding meaningful load to kube-apiserver. Writes work differently: r.Update() and other mutations go straight to the API server, bypassing the cache entirely, which means a Get() called immediately after an Update() can return stale data until the corresponding watch event arrives. The cache is eventually consistent, not strongly consistent.

What This Means for You

The tradeoff that matters most in practice is memory, not staleness. Cache size scales directly with the size and number of objects being watched and the indexes configured on top of them, and it’s easy for that footprint to grow into gigabytes on a large cluster without anyone noticing until a controller pod starts getting OOMKilled — a version of the resource-overhead problem Stack Archive has covered before in the context of virtual clusters’ hidden platform tax. The second trap is a List() call written without proper indexing: instead of using the cache’s indexes for a targeted lookup, an unindexed List() degrades into an O(n) linear scan across every cached object of that type, which can mean tens of thousands of objects on a busy cluster turned into a full scan on every reconcile. Both failure modes are invisible on a small test cluster and only show up at real scale, which is exactly when they’re most expensive to debug. For the narrow set of cases where eventual consistency genuinely isn’t acceptable — a check that must see the write just made, not a cached view from a few hundred milliseconds ago — controller-runtime exposes an APIReader that bypasses the cache and queries the API server directly, but it’s meant to be the exception, not the default read path.

Source: Kubernetes Blog

Content created with AI assistance and reviewed for accuracy.