Lessons from Running 1 Million Lambda Functions: What Scale-to-Zero Actually Costs
Cloud Platform Engineering“Scale to zero” sounds like a cost promise. At one million Lambda functions, ProGlove discovered it’s also an operational discipline problem that doesn’t reveal itself until you’re already in production.
How It Works
ProGlove built a fully serverless, multi-account SaaS platform on AWS Lambda — one account per tenant, providing quota isolation so no single tenant affects others — and scaled it past one million functions. Three failure modes emerged that don’t appear at smaller scales:
The synchronized scheduling attack. Every tenant backend used the same rate(5 minutes) EventBridge expression. At scale, thousands of Lambda invocations fire at exactly the same clock minute. The resulting burst looks like a self-inflicted DDoS attack on downstream services. Fix: introduce jitter, randomized batch offsets, and staggered scheduling. Principle: never do the same thing at the same time everywhere.
Observability cost explosion. Multi-account logging appeared manageable at $3/account/month in early testing. At scale, forwarding all observability data to a central account nearly doubled the cloud bill. The team restructured to forward selectively — only what SOC needed for correlation and alerting — rather than centralizing everything. The lesson: observability centralization decisions need to be made at design time, not at invoice time.
The SQS idle polling trap. Standard resilience patterns insert SQS queues between Lambda invocations. Each queue requires continuous polling even when empty. At one million functions, the idle polling cost destroys the scale-to-zero value proposition entirely. ProGlove replaced per-account DLQ polling with a centralized Dead Letter Queue model, reducing idle cost to approximately $0.70 per account per month.
What This Means for You
These failure modes appear well before one million functions. The synchronized scheduling problem can surface at a few hundred tenants. Observability cost implosion hits the moment you centralize multi-account logging without a filter layer. The SQS idle trap affects any team that added queues for resilience without modeling what continuous polling costs at their expected function count.
Additional scale decisions that proved right: account-per-tenant isolation for quota separation, direct EventBridge-to-Lambda to eliminate SQS intermediaries where resilience buffering isn’t required, and a mono-repo strategy to enforce consistent security scanning and configuration across all functions. Configuration drift at scale is a security and operational reliability problem, not just a maintenance nuisance.
ProGlove also found that CloudFormation StackSets hit undocumented performance ceilings at this scale, requiring custom deployment tooling — and their collaboration with AWS on that problem influenced platform improvements. Standard infrastructure-as-code tooling has scale ceilings; identify yours before you hit them in production.
The So What
The cost to add jitter to a scheduling expression is a few minutes. The cost to discover synchronized scheduling in production is an incident. The DLQ centralization pattern is worth implementing at ten tenants — by one hundred, it’s non-optional. If your serverless architecture works fine today, these failure modes will find you before you find them, unless you build jitter and selective observability forwarding into your design before the scale that makes them matter.
Source: AWS Architecture Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →