Datadog's MCP Security Agents: What Production Looks Like Beyond the Demo

Most MCP tool demonstrations show an agent running a query. Datadog’s Cloud SIEM team went further: they shipped tools to production, measured how users actually interact with agents, and published the architectural decisions that kept those tools from degrading existing security workflows. The result is a rare look at what production-quality agentic security tooling actually requires.

How It Works

Datadog built an MCP server for Cloud SIEM centered on three tools: a detection rule schema tool, a signal investigation tool, and a bulk triage tool. Each was designed around observed user behavior rather than assumed use cases. The team analyzed API logs, RUM data, and internal AI chat logs before writing a line of tool code — and found that 44% of user interactions involved detection rule authoring, with significant demand for signal investigation and bulk triage operations.

That data-first design process determined the tools’ scope. Bulk triage was redesigned to accept search queries instead of signal ID lists — a single call can now process up to 500 signals, compared to the 50-signal UI limit that 25% of customers were regularly hitting. One customer triaged 350+ signals in nine minutes.

The critical architectural constraint is MCP’s shared context window. When multiple security products contribute tools, each new tool reduces the reasoning space available to the agent. Datadog’s response is progressive disclosure: tools return only the schema sections relevant to the specific task, filtering by product type and detection method. Token usage dropped 41–47% compared to full schema delivery.

What This Means for You

The technical decisions surface three reusable patterns for teams building MCP tools in complex security domains.

Data over assumptions. Tool features built on usage analysis outperform features built on product intuition. The bulk triage redesign didn’t come from an engineer reasoning about scale — it came from observing that 14% of customers regularly hit the 50-signal ceiling. User behavior data is more reliable than intuition when designing for agent workflows, because agent interactions diverge from UI interactions in ways that aren’t obvious in advance.

Context budget is a first-class design constraint. In multi-product MCP environments, context is scarce. Progressive disclosure is the pattern: return what’s needed for the task at hand, not everything the agent might conceivably want. A 41% token reduction isn’t just an efficiency gain — it changes what the agent can reason about within a single interaction window.

Non-deterministic systems need custom evaluation infrastructure. Datadog built a testing framework that mocks tool responses to reduce cost, measures tool selection accuracy (weighted most heavily in scoring), and flags regressions exceeding 5% from baseline. Standard deterministic test frameworks aren’t designed for systems where the “correct” output varies. Teams shipping agentic tools need to invest in eval infrastructure before GA, not after the first regression surfaces in production.

Source: Datadog Engineering Blog

Content created with AI assistance and reviewed for accuracy.