What Running Kubernetes at Million-Cluster Scale Taught AWS About Zonal Failures

Amazon EKS runs hundreds of thousands of Kubernetes clusters across more than thirty AWS regions. Operating at that scale produces lessons that are hard to derive from smaller environments — and the zonal failure pattern is one of the clearest.

The lesson isn’t about which component failed. It’s about what the system did when it detected the failure.

What Happened

The AWS EKS team published a deep-dive on zonal resilience — the engineering behind EKS Zonal Shift and the production incidents that shaped it. The core insight from operating at this scale: most availability problems don’t come from a component failing. They come from a component reacting to a problem in a way that makes it worse.

Control planes that attempt to rebalance during a zonal impairment. Schedulers that trigger cascading evictions across healthy zones. Auto-scaling policies that interpret degraded zone capacity as demand and scale aggressively into unhealthy infrastructure. Each reaction is individually logical; collectively, they convert a contained zonal issue into a cluster-wide event.

The team’s response to this pattern is what they call static stability: during a zonal impairment, the most valuable action a system can take is to stop reacting, preserve existing capacity in healthy zones, route traffic around the affected zone, and wait.

What EKS Zonal Shift Does

ARC Zonal Shift is the AWS implementation of the static stability principle at the cluster level. It allows operators to move workloads out of an impaired Availability Zone in a controlled manner — redirecting traffic, not triggering workload eviction in the healthy zones.

Key behavior: Zonal Shift is designed as a temporary measure. It runs until the AZ recovers or until the operator cancels it. The design prioritizes not touching what’s working while giving the impaired zone time to stabilize.

In May 2026, AWS added Karpenter support for ARC Zonal Shift — meaning Karpenter-managed node provisioning now respects active zonal shifts, preventing new nodes from being scheduled into impaired zones during a shift.

Why This Matters for Platform Teams

The reaction problem is harder to see than the failure problem. Most runbooks describe what to do when a component fails. Fewer explicitly address what automated systems should stop doing during a zonal event. The EKS experience suggests that constraining automated reaction may be as important as the response action itself.

The pattern generalizes beyond AWS. Static stability as a principle — preserve what’s working, route around what isn’t, suppress rebalancing reactions — applies to any multi-zone Kubernetes deployment regardless of cloud provider. The EKS implementation is specific to AWS; the failure pattern it addresses is not.

Automation detection is critical. The AWS team’s practical recommendation: automate detection and response to zonal failures. Don’t wait for the cluster to degrade before acting — by the time degradation is obvious, cascading reactions may already be in progress.

What This Means for Builders

Audit your automated reactions during zonal events. HPA policies, cluster autoscaler behavior, and node rebalancing all have failure modes under zonal impairment. Map what each automation does when it detects degraded zone capacity, and verify that the response doesn’t amplify the problem.

Enable Zonal Shift on EKS clusters where availability matters. The feature is available but opt-in. If you’re running production workloads across multiple AZs and haven’t evaluated ARC Zonal Shift, the EKS team’s documentation is now the best starting point.

Test with intentional zone impairments. The static stability principle is easy to understand and hard to verify without actual chaos testing. Scheduled zonal impairment exercises — even at small scale — surface automation reaction issues that won’t appear in standard load tests.

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.