How 121 Million gRPC Connections Exposed a Fatal DNS Assumption

When 121 million mobile devices established persistent gRPC connections at the start of a T20 World Cup broadcast, bitdrift’s infrastructure hit a 79.8 percent peak error rate — despite having more than enough NLB capacity to handle the load. The root cause was not hardware. It was a DNS routing policy nobody had validated against a synchronized mass-connection event.

What Happened

Bitdrift is a mobile observability platform that streams real-time telemetry over long-lived gRPC connections from devices in the field. During the T20 World Cup, their customer experienced a flash crowd: traffic surged from near-zero to over 110,000 peak requests per second the instant the broadcast went live. The standard CloudFront → Route 53 → Network Load Balancer → EKS architecture was in place, with NLBs distributed across three availability zones. Adding more NLB capacity had no effect.

An investigation traced the failure to Route 53’s weighted routing policy. At each DNS resolution, the policy returned a single IP address to the requesting CloudFront edge node. With persistent gRPC connections, each edge node resolved DNS once at broadcast start, received that same IP, and held the connection for the broadcast duration. Instead of load distributing across all NLBs over successive TTL windows, every new connection at broadcast start funneled to whichever single origin the DNS query returned — a thundering herd that overwhelmed the selected NLB while the others sat underutilized.

Why It Matters

The incident surfaces a foundational mismatch between stateless HTTP assumptions and persistent connection protocols. In conventional HTTP workloads, short-lived connections naturally distribute over time as DNS TTLs expire and successive queries return different answers from a weighted pool. gRPC and WebSockets invert this dynamic: connections are long-lived, DNS is resolved once per connection, and the distribution of load becomes a function of the establishment pattern rather than ongoing traffic. At moderate concurrency, the failure mode is invisible. At the scale of a synchronized broadcast start — millions of devices reconnecting within seconds — weighted routing behaves as a single point of failure.

The resolution required no code changes. Migrating from weighted to multi-value answer routing causes Route 53 to return up to eight IP addresses per query with built-in health checks. CloudFront edge nodes can immediately distribute new gRPC connections across the full NLB fleet from the first DNS resolution. Error rate at the next live event: zero.

What This Means for Builders

Platform teams running gRPC or WebSocket workloads behind CloudFront have almost certainly inherited a weighted routing configuration — it is the default, it works fine at ordinary concurrency, and it passes standard load tests. The failure mode only surfaces under two conditions that rarely co-occur in testing: a persistent connection protocol and a synchronized mass-connection event. Broadcast starts, flash sales, and mass push notifications triggering simultaneous app opens can all produce this pattern. Standard load tooling does not replicate genuinely synchronized bursts at broadcast scale, so the gap between testing and production remains invisible until an incident.

The interval between provisioning more NLB capacity and seeing no improvement was the diagnostic signal that the problem lived upstream of the origin. That signal is worth recognizing: if adding capacity at the origin does nothing, the routing layer is worth examining before anything else.

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.