Kubernetes Didn't Fix Databases. It Just Hid the Problem

Kubernetes made deploying software easy. Run a Helm install, wait a minute, and Postgres, Redis, or OpenSearch is up and running. Most platform teams have quietly assumed that ease extends to running those databases too. It doesn’t.

Why Everyone Believes This

The belief traces back to the DevOps movement’s founding maxim: “you build it, you run it.” For close to two decades, that principle collapsed the wall between developers and operations, and Kubernetes looked like its natural extension for data services. If a team can helm install a stateful application in minutes, the reasoning goes, that same team should be able to operate it. The deployment experience is genuinely simple, and it’s easy to mistake a simple deploy for a simple system. Nothing about spinning up a database on Kubernetes signals that patching, backups, and failover sit in an entirely different skill category than shipping application code.

What the Evidence Shows

The gap shows up first at upgrade time, not deploy time. A single-node Postgres instance is trivial to patch. But avoiding downtime means running a cluster, typically three nodes, so one can take over if another fails, and upgrades on that cluster have to happen one node at a time. That’s where things go wrong: if the leader election that decides which node takes over fails, two nodes can end up believing they’re in charge simultaneously, producing inconsistent writes and, in the worst case, outright data corruption, according to an account from a senior cloud platform engineer writing in The New Stack. Sometimes a restart and re-election resolves it in minutes. When the data itself is corrupted, the outage can stretch into hours, and the business impact is concrete: a hotel booking platform whose database goes down can’t take reservations, and if backups weren’t actually working, it can lose the bookings outright, with no record of who’s arriving next week.

Staffing doesn’t fix this either. The U.S. Bureau of Labor Statistics projects just 4% growth for database administrators and architects from 2024 to 2034, with most job openings coming from replacing departing workers rather than genuine new supply. Even where specialist hires exist, one is never enough. Database failures happen at night, so a real on-call rotation is required, and different databases (Postgres, Redis, OpenSearch) need different specialists, meaning teams end up running several parallel on-call schedules rather than one.

The Correction

Kubernetes solved deployment. It did not solve operations, and treating the two as equivalent is the actual mistake. The workable model isn’t “developers run their own databases because Kubernetes made it possible” — it’s developers requesting databases through automation while a platform team, not the application team, owns patching, backups, and failover behind the scenes. Good database operations are supposed to be invisible: patches applied on a schedule, backups that are tested and provably restorable, and a developer’s involvement limited to three moments — requesting a new database, needing credentials, or triggering a backup ahead of a risky release. Everything else is a platform-team problem, not an application-team problem. This is the same pattern behind Kubernetes’ other hidden operational costs: the parts of the system that don’t show up in a demo are exactly the parts that eat platform engineering time at scale.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.