CloudFormation's Custom Resources Have No Failover Story. AWS Just Wrote One.
Cloud Platform EngineeringCloudFormation custom resources let teams extend infrastructure-as-code to anything AWS doesn’t natively support — third-party APIs, custom database initialization, cross-account orchestration. What they’ve never had is a real answer for what happens when the region running that Lambda handler goes down.
How It Works
Custom resources depend on a Lambda function to do the actual provisioning work behind a CloudFormation-managed piece of infrastructure. AWS’s own architecture teams identify four structural gaps in that setup: there’s no native fan-out mechanism to run the same handler across multiple regions, deploying handlers redundantly creates a duplicate-execution risk, there’s no built-in distributed locking to guarantee only one region actually processes a given event, and failover between regions has to be built and triggered manually. Idempotency — making sure a retried or duplicated event doesn’t double-provision a resource — has always been the individual developer’s problem to solve, not the platform’s.
The reference architecture answers all four at once with an active-active design running in two regions simultaneously (the example uses us-east-1 as primary and us-west-2 as secondary). Every event publishes to both regions via Amazon SNS fan-out, landing in a dedicated SQS queue in each. The primary region’s queue processes immediately; the secondary’s queue holds the same event back using a configurable visibility delay, so it only picks up processing if the primary hasn’t already handled it. DynamoDB Global Tables provide the distributed lock and idempotency ledger, replicating lock state bidirectionally between regions so whichever region gets there first is authoritative — the other backs off automatically rather than double-processing. Amazon Application Recovery Controller watches for regional failure and triggers the actual failover, with CloudWatch handling monitoring and alarms across the whole setup.
What This Means for You
The pattern matters most for exactly the use cases custom resources tend to get used for: third-party API calls during stack provisioning, database initialization and secret generation, cross-account orchestration, and compliance checks baked into deployment — all cases where a silent regional outage mid-deployment can leave infrastructure half-provisioned with no clear record of what happened. Teams currently running custom resources single-region are, by definition, accepting that failure mode today. This architecture is a documented, AWS-sanctioned way to stop accepting it, using services (Lambda, DynamoDB Global Tables, SNS, SQS, Application Recovery Controller) most teams already have in their AWS toolkit rather than a net-new platform to adopt.
The So What
If a custom resource handler is doing anything you’d consider mission-critical — third-party integrations, secret generation, compliance gates — single-region deployment is a liability you’re carrying without a documented reason to. This active-active, DynamoDB-locked pattern is the reference answer AWS itself is pointing teams toward; treat it as the default posture for custom resources rather than an advanced option to get to later.
Source: AWS Architecture Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →