The 5-Step Framework for Automated Root Cause Analysis at Cloud-Native Scale

Manual root cause analysis is a correlation problem — and at scale, correlation is not something humans should be doing in the middle of an incident. Atlassian’s published framework for multi-signal RCA shows a structured path from “hundreds of services” to “ranked hypotheses in minutes,” using OpenTelemetry as the common data substrate.

The Framework

The core insight behind Atlassian’s approach is that human engineers are slow at finding signal correlations across telemetry types, but fast at validating a small candidate set. The framework inverts the work: automate correlation to generate a ranked list, then hand that list to engineers for validation and resolution.

Atlassian runs hundreds of interconnected microservices across multiple regions. During production incidents, the telemetry volume is overwhelming — every service emits metrics, traces, and logs simultaneously. The system narrows this to “tens of services rather than hundreds” before a human looks at anything.

Step 1: Scope the Search

Rather than analyzing all services simultaneously, the system first builds a service dependency graph from OpenTelemetry spans. This graph represents causal relationships between services — which services call which, and in what direction. When an incident is detected, the initial search scope is the affected service plus its first- and second-degree dependencies.

This step alone eliminates most of the irrelevant signal. A database latency spike doesn’t warrant inspecting the frontend rendering layer.

Step 2: Independent Anomaly Detection per Signal Type

Each signal type — metrics (using RED signals: Rate, Error rate, Duration), distributed traces (structural anomalies, exceptions, latency spikes), and logs (semantic clustering for novel error patterns) — runs anomaly detection independently. Combining signals too early introduces noise; independent detection ensures each type surfaces its own anomalies cleanly.

Step 3: Temporal Correlation

Anomalies detected across signal types are grouped by a ±5 minute sliding window. Co-occurring anomalies across metrics, traces, and logs within this window are treated as candidate components of the same incident. Deduplication uses sequence fingerprinting — repeated failure patterns collapse into a single hypothesis rather than generating redundant entries.

Step 4: Graph-Based Causal Analysis

The grouped anomalies are then evaluated against the service dependency graph from Step 1. This step adds causal direction: an anomaly in Service A that depends on Service B is evaluated differently than an anomaly in Service B that nothing depends on. The graph provides the hypothesis direction — upstream faults propagating downstream vs. isolated failures — that pure correlation cannot.

Step 5: Hypothesis Generation with Confidence Scoring

The output is a ranked list of hypotheses, each scored using a formula combining temporal cohesion (how tightly co-occurring the anomalies are) and causal path strength (how direct the dependency path is). Crucially, scores alone are insufficient — the system generates human-readable narratives with telemetry references, allowing engineers to validate the hypothesis without re-querying the raw data.

The Atlassian team notes that explainability matters as much as accuracy for production trust. A confidence score the on-call engineer can’t interpret is operationally useless.

Where It Doesn’t Apply

This framework assumes a well-instrumented OpenTelemetry data layer. Organizations without distributed tracing at the span level cannot build the dependency graph in Step 1, which collapses the whole approach to standard correlation-by-metric — a much weaker signal. The framework also requires the team to validate, not just consume, the hypotheses; if hypothesis generation becomes a checkbox rather than an engineering signal, the production trust benefit disappears.

Source: Cloud Native Computing Foundation

Content created with AI assistance and reviewed for accuracy.