Datadog Cut Spark Compute Costs 44% with an AI Agent. Here's Exactly How.

$250,000 in annual savings from three lines of changed Scala code. That’s the return Datadog’s internal platform team got from wiring an AI agent to their Spark jobs observability stack — and the methodology is worth understanding.

The Number

Datadog’s ServiceQueryEdge job builds a knowledge graph mapping relationships between observability entities. It runs daily across seven datacenters, processing up to 27TB of input and 16 billion records per partition.

Before optimization:

• Duration: 17+ hours per run
• Daily cost: $1,500 (US1 datacenter)
• Peak memory: 3.12TB
• Task duration P99: 2h 55m

After three AI-identified optimizations:

• Duration: ~7 hours (60% reduction)
• Daily cost: ~$830 (44% reduction, approximately $250K annual savings in US1)
• Peak memory: 2.19TB
• Task duration P99: 1h 13m
• Executor time: -50% in US1

How the Math Works

The team built an AI agent using Claude that ingested Jobs Monitoring telemetry — stage metrics, SQL execution plans, and distributed trace data — and correlated slow operators back to specific source code sections.

The agent identified three optimization opportunities that would have required hours of manual investigation to find individually:

1. Removed a redundant distinct() call — an aggregation step that was forcing an unnecessary shuffle operation, scattering data across the cluster before recombining it. Eliminating it reduced inter-node data movement substantially.

2. Replaced SortMergeJoin with BroadcastJoin — the job was sorting 7.4 billion rows to perform a join, when one side of the join (628 metrics) was small enough to broadcast to all executors instead. BroadcastJoin eliminates the sort entirely and reduces the join to a lookup.

3. Fixed improper salting logic — the job was using modulus-based key salting, which inherited the skew from the original data distribution. Switching to rand() salting produces uniform distribution, eliminating the slow task stragglers that inflated P99 duration.

Each fix was a targeted change to the Scala codebase. None required infrastructure changes or scaling up compute.

The agent also used a subagent architecture: a primary agent identified bottlenecks, while a validator subagent checked each proposed fix for contraindications before implementation — filtering false positives before they reached the engineering team.

What It Signals

This case study is notable for three reasons. First, the agent worked from observability data, not code inspection alone — it connected Jobs Monitoring telemetry to execution plans to source code, which is the kind of multi-signal correlation that takes human analysts significant time. Second, the fixes were code-level, not infrastructure-level — no new clusters, no instance type changes, just more efficient logic. Third, the validator subagent pattern represents a practical approach to AI-generated recommendations in production: generate broadly, validate narrowly.

For platform teams evaluating agentic AI for cost optimization, this represents the highest-fidelity evidence available: a published case study from a vendor that uses its own tools in production, with specific before/after metrics and code-level change descriptions.

Source: Datadog

Content created with AI assistance and reviewed for accuracy.