How Datadog Used AI to Turn a 45-Minute Query Into a 1-Second One

Migrating a live routing system that processes over 100 trillion events a day is not an obvious place to experiment with AI-assisted code generation. Datadog’s engineering team did it anyway, and the production numbers make the case.

What Happened

Datadog’s Stream Router is the internal control plane that determines which Kafka cluster, topic, and partition receives each metrics datapoint flowing through the company’s observability platform. The system’s storage layer — a FoundationDB key-value store — had hit structural limits: routing tables had grown to the point where bulk update operations took approximately 45 minutes due to transaction size constraints, and the relational data that the application needed was being reconstructed in application code from a flat KV model.

The migration target was a properly normalized PostgreSQL schema, with DuckDB handling read-side static snapshots. The engineering team used Claude and Cursor through a structured three-phase workflow: documentation generation (using Claude to produce intent-focused documentation per function), a stub-test-iterate loop (create stub returning error, run tests, feed failures + old code + schema context to AI model, iterate until passing), and a blue/green production validation that ran both storage backends in parallel for several weeks.

The results: operations that took approximately 45 minutes now complete in approximately 1 second. Latency dropped from hundreds of milliseconds to single digits. CPU and memory consumption decreased across pods. Database expenses dropped 90 percent after retiring the managed FoundationDB clusters. The migration covered a dataset that shrank 40x in PostgreSQL versus the KV representation. Total duration: three months, with the core proof-of-concept in four weeks. Zero incidents during cutover.

Why It Matters

Datadog’s write-up is valuable specifically because it’s honest about what worked and what didn’t. AI-generated SQL was frequently correct but unoptimized — requiring human input for batching, UNNEST tricks, and common table expressions. High token consumption from verbose test output was a recurring friction point. The lesson the team drew: “The quality of your test suite is the ceiling for how much you can trust AI-generated code.”

The stub-test-iterate loop is worth examining as a replicable pattern. Rather than asking an AI model to migrate an entire function or module at once, the approach creates a method stub, immediately runs the test suite against it, and feeds the specific failure output alongside the original implementation and target schema to the model. The binary success criterion — tests pass or they don’t — removes the ambiguity from evaluating AI output at each step.

Narrow prompts outperformed broad context prompts. Well-commented code explaining the why of logic (not just the what) produced better AI output. Fresh context sessions when token counts grew too large — with the documentation generated in phase one serving as external memory — maintained output quality throughout a three-month engagement.

What This Means for Platform Teams

The migration pattern Datadog describes isn’t novel in its phases — documentation, implementation, validation is standard engineering practice. What changes with AI-assisted refactoring is the speed and cost of the implementation phase, specifically for the kind of rote transformation work that makes migrations tedious: reconstructing relational semantics from flat storage models, translating application-layer logic into database-native SQL, and handling the edge cases that accumulate across a large API surface.

The prerequisite for this approach is a test suite with genuine coverage of production behavior. The team’s framing — test quality as the ceiling for AI trust — is the structural constraint that determines whether AI-assisted migration is viable for a given system.

Source: Datadog Engineering Blog

Content created with AI assistance and reviewed for accuracy.