The Framework for Zero-Downtime Kubernetes When a Whole Cluster Dies

When a single Kubernetes cluster goes down, failover is usually a runbook — a human-triggered sequence of DNS changes, traffic shifts, and hoping nothing was missed. A CNCF-documented test using Linkerd’s multicluster extension shows that failover doesn’t have to be a runbook at all; it can be an automatic rebalance that happens before anyone gets paged.

The Framework

Linkerd’s multicluster extension lets several Kubernetes clusters present the same services as unified, load-balanced endpoints across a full-mesh network. The organizing idea is that once clusters are meshed together, “the same service” can mean multiple physical deployments in different regions transparently — and the failure mode changes from “a cluster died, now what” to “traffic already moved, deal with the postmortem later.” The CNCF test deployed three GKE clusters across regions, each running three e2-medium nodes with non-overlapping pod CIDRs, connected via VPC peering.

Federated Mode

The default and simplest mode: same-named services across clusters merge into a single load-balanced endpoint automatically, with no application code changes required. A client calling the service has no idea whether the request landed in the local cluster or was routed to a healthy peer. This is the mode the CNCF test used for its zero-downtime demonstration — three clusters exposing one logical service.

Flat Mirroring

Services are also exposed individually as <service>-<cluster>, with traffic routed pod-to-pod directly rather than through federation. Use this when you need to deliberately target a specific cluster’s instance of a service — for staged rollouts or debugging a single region — without giving up the ability to fall back to federated routing elsewhere.

Gateway Mirroring

Cross-cluster traffic routes through a gateway rather than requiring flat, direct pod networking between clusters. This mode is the answer when clusters can’t have fully flat networking between them — a common constraint in multi-cloud or multi-account setups where VPC peering across every cluster pair isn’t practical, similar to the constraint Azure Kubernetes Fleet Manager’s Cilium-based cross-cluster networking was built to address on a different mesh stack.

Where It Doesn’t Apply

The chaos test is the part worth taking seriously before adopting this pattern: when one cluster was scaled to zero replicas, federated traffic rebalanced automatically from an even 33/33/33 split across the three clusters to 50/50 across the two survivors, with zero errors recorded. But this is a demonstration at three e2-medium nodes per cluster and roughly $10-15/day in infrastructure cost — the failure modes that matter at production scale (cross-region latency during rebalance, mesh control-plane overhead, cost at real traffic volumes) aren’t the ones a three-node proof of concept surfaces. Treat the chaos-test result as evidence the pattern works, not as a substitute for testing it against your own traffic shape.

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.