The Quorum Math Behind Multi-Cluster Kubernetes Databases

A Kubernetes cluster is supposed to be the failure domain everything else survives — except the database running inside it, which goes down as one unit right along with the cluster. A regional outage, a corrupted control plane, or a network partition takes the whole database with it, and there is no automatic recovery path, because nothing about a single-cluster deployment was built to have a “next cluster” to fail over to.

How It Works

A multi-cluster database architecture, as CNCF lays out using the Percona Operator for MongoDB, splits the deployment across clusters by role rather than just by location. One cluster is the Main Site: fully managed by the operator, holding the Primary node, provisioning TLS certificates and user credentials, and handling writes. A second cluster is the Replica Site: the operator there runs in “unmanaged mode” — it doesn’t generate its own certificates or try to initialize a new replica set, it joins the one the Main Site already created. Secrets get copied from Main to Replica so both sides authenticate against the same identity.

The topology that makes this survivable is what CNCF calls the 2+2+1 pattern: two voting nodes in Cluster A, two in Cluster B, and one non-voting member in a third location. The design principle is deliberately an odd number of voting members spread across locations, because electing a Primary requires a strict majority — 3 of 5 votes in this case. That arithmetic is the entire point: with an even split, a network partition can leave both sides holding half the votes and neither able to elect a Primary. An odd, cross-cluster vote count is what prevents that deadlock.

None of this works without cross-cluster node discovery, which is where the Multi-Cluster Services (MCS) API comes in. MCS creates ServiceExport and ServiceImport resources that publish services into a shared DNS zone (svc.clusterset.local), letting a pod in one cluster resolve and reach a pod in another. It requires its own implementation underneath — Submariner, Cilium ClusterMesh, or a managed provider’s native fleet networking — and the resources take roughly five minutes to synchronize across clusters once created. Pods also have to be exposed as type: ClusterIP for MCS to see them at all.

What This Means for You

The failure mode this architecture is built for is real but not free. After a regional loss, the surviving side runs in what CNCF describes as a degraded state — fewer resources than normal, and no more safety margin, since losing one additional node past that point pushes the deployment into read-only mode. Cross-cluster latency also stretches the election window beyond its roughly 12-second median default (tunable via electionTimeoutMillis), which means applications need retryable writes to ride out the gap where there’s briefly no Primary at all, rather than surfacing that gap as an outage to users.

The operational cost is running two independent Kubernetes control planes that must never disagree about which side owns the data — avoiding a split-brain scenario takes careful operator configuration and, in some recovery paths, manual replica set finalization rather than anything fully automatic. For relational workloads instead of MongoDB, the same pattern shows up in projects like Vitess and CloudNativePG, which solve the same cross-cluster quorum and discovery problem for their respective data models. Teams already running cross-cluster networking at the platform layer have a head start here, since MCS-style service discovery is the same primitive that pattern depends on — the database layer just adds the quorum math on top.

Source: CNCF

Content created with AI assistance and reviewed for accuracy.