Datadog Database Investigator: Root Cause Analysis Without a DBA

Database performance investigations are some of the most expensive escalations in platform engineering — requiring specialized expertise that most on-call engineers don’t have. Datadog’s Database Investigator is designed to run those investigations autonomously.

The DBA Bottleneck in Modern Platform Teams

Database performance incidents follow a predictable pattern: a latency spike surfaces in APM, the on-call engineer looks at dashboards, cannot identify the root cause without deep database expertise, and escalates to a database administrator who may not be available for hours. Connection pool exhaustion is worse — the visible symptom (new connections hanging) hides the actual cause (idle-in-transaction sessions consuming slots), and diagnosing it requires correlating connection state breakdowns, transaction durations, and wait events simultaneously.

Database Investigator, currently in Public Preview for PostgreSQL and Private Preview for Microsoft SQL Server, applies an agentic approach to this problem. Rather than presenting dashboards for a human to interpret, it examines workload metrics, query samples, execution plans, and logs independently, then provides root cause analysis with specific remediation steps and linked queries. Engineers can ask follow-up questions to refine the analysis — similar to how Datadog’s Bits AI conversational interface works for broader observability queries, applied specifically to the database layer.

Three Diagnostic Classes It Handles

Latency spikes: Database Investigator correlates distributed traces, query metrics, and execution plans to identify performance regressions. In Datadog’s documented example, it detected a query latency increase from 15ms to 447ms and identified the root cause: a query-level regression caused by an index scan flipping to a sequential scan on a large table. That diagnosis requires reading execution plans alongside trace data simultaneously — a correlation that on-call engineers without DBA background typically cannot make unaided.

Connection pool exhaustion: The tool analyzes connection state breakdowns, transaction durations, and wait events together, surfacing signals like idle-in-transaction sessions holding connection slots that other connections are queued behind. A human doing this investigation manually cycles through multiple dashboards and database views; the Investigator correlates them in a single pass with a natural-language explanation of the finding.

Replication lag: Database Investigator traces lag across entire database clusters, identifying specific queries and services blocking write-ahead log (WAL) replay and recommending targeted fixes. For teams running read replicas or multi-region PostgreSQL deployments, replication lag analysis previously required direct database access that most incident responders don’t have.

The tool integrates with Datadog’s broader Bits AI platform — the same agentic infrastructure layer that powers the Datadog Governance Console’s policy enforcement and Datadog’s security automation capabilities. Database Investigator extends that pattern into the data layer, treating database performance as an observability domain rather than a separate discipline requiring specialist escalation.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.