How to Test SQS Resilience Before an Outage Does It for You

Most SQS resilience configurations are never validated under real access failures. The circuit breaker that should trip after 30 seconds of connection failures — does it actually trip? The fallback storage path — has anyone confirmed it writes correctly when SQS is unreachable? The AWS Architecture Blog lays out a four-phase fault injection approach that turns these questions into observable results before a production outage provides the answer instead.

The Core Technique: Scoped Access Denial

The experiment denies SQS data-plane access while leaving management-plane actions available. This is the critical design choice: a broad deny policy that blocks all SQS operations would prevent the automation from cleaning up after itself. Denying only the operations the application uses — SendMessage, ReceiveMessage, DeleteMessage — while keeping GetQueueAttributes and similar management calls available, scopes the failure precisely.

The IAM-based deny policy is applied by a Systems Manager Automation document that can also remove it. This means the experiment is self-contained: the test infrastructure can restore normal access without requiring manual intervention, even if the primary automation has already moved on.

The Four-Phase Escalation

The experiment runs in four sequential impairment windows separated by recovery periods:

Phase 1 (2 minutes): Short disruptions reveal whether failure-handling mechanisms activate at all. Circuit breakers that need sustained failure to trip often miss this window entirely.

Phase 2 (5 minutes): The defined hypothesis threshold. A well-specified hypothesis looks like: “When our order processing application loses access to SQS for 5 minutes, we expect the producer to open its circuit breaker within 30 seconds.” If the circuit breaker hasn’t tripped by the end of phase 2, the configuration needs revision.

Phase 3 (7 minutes): Extends beyond the hypothesis threshold to expose behavior at the boundary — what happens after the circuit breaker trips, whether fallback storage accumulates correctly, whether message aging starts becoming a problem.

Phase 4 (15 minutes): Systemic issues that only appear under sustained failure. Backlog stalling, message age growth, and consumer behavior during prolonged unavailability all show up here.

What to Instrument

The experiment monitors two distinct failure surfaces:

Producer side: Does the producer detect that SendMessage is failing? Does it open the circuit breaker? Is it writing to fallback storage correctly? Metric: NumberOfMessagesSent (drops during impairment). Application-level: circuit-breaker state transitions, error counts, fallback storage write rate.

Consumer side: What happens to messages already in the queue? Can the consumer still receive and process them? Does the backlog stall? Metrics: ApproximateNumberOfMessagesVisible (queue backlog), ApproximateAgeOfOldestMessage (tracks message aging as backlog ages).

Stop Conditions: Customer Metrics, Not Queue Metrics

The FIS experiment template uses CloudWatch alarms as stop conditions — but the alarms should monitor customer-impact metrics rather than queue metrics. An alarm on ApproximateNumberOfMessagesVisible would stop the experiment as soon as messages accumulate, which is the expected behavior during an access failure. An alarm on customer-visible error rate stops the experiment when real harm is occurring: failed orders, dropped events, timeout responses.

This distinction prevents the stop condition from interfering with the experiment while still providing a safety cutoff if the failure propagates beyond the SQS tier.

The Supporting Patterns

The AWS post names four practices that make this experiment type work:

Exponential backoff with jitter. Retries without jitter create correlated spikes when multiple clients retry simultaneously. Jitter spreads the retry load across the recovery window.

Circuit breakers. Required to prevent retry storms from overwhelming SQS when access returns. Without a circuit breaker, a 15-minute outage can generate 15 minutes of queued retries landing simultaneously.

Idempotent message processing. When SQS access is restored and the consumer resumes, it will reprocess messages that were enqueued during the outage. If message processing is not idempotent, the recovery itself causes duplicate-processing failures.

Dead-letter queue with appropriate retention. Messages that cannot be processed after N retries need a place to land. DLQ retention should be longer than the recovery SLA to prevent messages from expiring before they can be diagnosed.

Source: AWS Architecture Blog

Content created with AI assistance and reviewed for accuracy.