Blog · Topic

Incident Response

23 analyses tagged Incident Response
Cloud Platform Engineering

API Gateway WebSocket Failover: Keep Sessions Across Regions

API Gateway WebSocket failover breaks every live connection. AWS's multi-Region pattern restores sessions using a canary, Route 53 and Global Tables.

Cloud Platform Engineering

Google SRE Book Second Edition: What Changed in 10 Years

Google's SRE book second edition is a full rewrite for AI and platform engineering. What changed in SLOs, observability and team models, and who should read it.

Cloud Platform Engineering

The Closed-Loop Incident Response Pattern: How AWS DevOps Agent Uses OpenSearch MCP to Investigate Itself

OpenSearch alerts can hand AWS DevOps Agent the same observability data via MCP, taking incidents from SNS trigger to root cause analysis in five minutes.

Cloud Platform Engineering

Wiring AWS DevOps Agent to Jira, ServiceNow, and PagerDuty via EventBridge

How to route AWS DevOps Agent investigations to Jira, ServiceNow, or PagerDuty with EventBridge and Lambda: the full integration architecture.

Cloud Platform Engineering

Beyond the 30-Minute AZ Test: Running Multi-Day AWS Availability Zone Evacuation Drills

Multi-day AWS AZ evacuation drills expose failures short tests miss: capacity drift, DNS TTLs, pool exhaustion. A runbook for 72-hour ARC zonal shift drills.

Cloud Platform Engineering

The 5-Step Framework for Automated Root Cause Analysis at Cloud-Native Scale

Atlassian's published approach to multi-signal correlation shows how to correlate metrics, traces, and logs to generate ranked RCA hypotheses automatically — narrowing a 200-service search to tens of services within minutes of an incident.

Cloud Platform Engineering

AWS DevOps Agent + SageMaker HyperPod: Automated Incident Triage at ML Scale

HyperPod self-heals hardware faults automatically, but configuration issues and stuck pods still require 30-45 minutes of manual triage. AWS's DevOps Agent cuts that to minutes.

Cloud Platform Engineering

AWS DevOps Agent + Wiz: Same Alert, Three Different Incidents

AWS DevOps Agent now queries Wiz over MCP mid-investigation, turning an ambiguous CPU spike into a confirmed security incident, an operational fix, or a coverage gap.

Cloud Platform Engineering

CloudFormation's Custom Resources Have No Failover Story. AWS Just Wrote One.

AWS's reference architecture for multi-region CloudFormation custom resources solves duplicate execution and failover gaps native CloudFormation was never built to handle.

Cloud Platform Engineering

How 121 Million gRPC Connections Exposed a Fatal DNS Assumption

Bitdrift's T20 World Cup incident: a single DNS routing policy caused 80% errors when 121M devices connected at once. The fix required zero code changes.

Cloud Platform Engineering

AWS DevOps Agent + Kiro CLI: The Incident Remediation Loop That Fixes Itself

AWS has published a production blueprint where DevOps Agent investigates incidents and Kiro CLI applies the fix — reducing MTTR by 75% with a single human touchpoint: PR approval.

Cloud Platform Engineering

What Running Kubernetes at Million-Cluster Scale Taught AWS About Zonal Failures

AWS operates hundreds of thousands of EKS clusters across 30+ regions. The core lesson from that scale: most availability problems don't come from components failing — they come from components reacting in ways that make failures worse. Here's what static stability means in practice.

Cloud Platform Engineering

How S&P Global Hit 15-Minute RTO on Capital IQ Using Amazon FSx and NetApp ONTAP

S&P Global's Capital IQ platform now achieves sub-15-minute read-only failover using Amazon FSx SnapMirror and FlexClone pre-provisioning. The design turns disaster recovery into a traffic redirect.

Cloud Platform Engineering

Your AI Pipeline Is Broken and Your Dashboards Don't Know It

Traditional monitoring shows binary pass/fail. LLM pipelines fail probabilistically — silently returning plausible but wrong outputs while every health metric shows green. Here's what to instrument instead.

Cloud Platform Engineering

AWS DevOps Agent + LaunchDarkly: Feature Flags as Incident Response

AWS DevOps Agent now connects to LaunchDarkly's MCP server, enabling agentic feature flag orchestration at deployment and during incidents — flag-based containment in minutes, not rollbacks.

Cloud Platform Engineering

AWS DevOps Agent and Datadog MCP Server Are Both Generally Available — Here's What Changed

Both tools exit preview simultaneously: DevOps Agent now coordinates with Slack, PagerDuty, and ServiceNow; Datadog's MCP Server routes AI agent prompts to live observability data. Together they close the detection-to-prevention loop.

Cloud Platform Engineering

AWS Kiro IDE Now Integrates DevOps Agent: Incident Response Without Leaving Your Editor

The AWS DevOps Agent integration in Kiro brings CloudWatch metrics, X-Ray traces, deployment history, and cost optimization directly into the IDE — collapsing the context switch between writing code and investigating production issues.

Cloud Platform Engineering

AWS Extends Its DevOps Agent to Diagnose EKS Node Issues via Custom MCP

A custom MCP server bridges AWS DevOps Agent to SSM Automation runbooks, giving the agent structured access to 20+ OS-level log sources — and it completed an autonomous DNS root-cause analysis in 14 minutes.

Cloud Platform Engineering

82% of Observability Alerts Are Never Acted On. The Data Shows Why.

PagerDuty data shows 82% of enterprise alerts are noise—while 41% of SREs have considered leaving over alert volume. The problem isn't more observability.

Cloud Platform Engineering

How Google SRE Is Using Agentic AI to Improve Operations

Google SRE deployed agentic AI across reliability design, anomaly detection, incident management, and autonomous mitigation — with governance principles for production-safe deployment.

Cloud Platform Engineering

AWS Resilience Hub's Next Generation Brings AI-Powered Failure Mode Analysis to SRE

AWS Resilience Hub now uses generative AI to automatically identify failure modes against defined policies. Here's what changed and why it matters for platform engineering teams managing production resilience at scale.

Cloud Platform Engineering

Only 37% of Developers Trust AI for Incident Response. The Gap Is Solvable.

PagerDuty's State of AI-First Operations survey reveals a 22-point trust gap between IT leaders and developers on AI in incident response — while $300K/hour outages fuel developer burnout at 42% of organizations.

Cloud Platform Engineering

AWS DevOps Agent: Autonomous Incident Response in Under 5 Minutes

AWS DevOps Agent uses agentic AI on Bedrock AgentCore to resolve incidents autonomously — from alarm to root cause analysis in under 5 minutes.

Newsletter coming soon.

Curated digests across AI, biohacking, photography, travel, and more. Be the first to know when we launch.