Datadog's BewAIre: From 97.4% to 99.86% Accuracy on Malicious Package Detection

97.4% accuracy sounds good until you’re scanning millions of packages. At that scale, the 2.6% error rate means tens of thousands of false positives or missed malicious packages. Datadog’s BewAIre system pushed that to 99.86% — and the architecture that makes it possible is worth understanding.

The Number

BewAIre detects malicious code in npm and PyPI packages with 99.86% accuracy on confirmed malicious packages, improved from an initial 97.4%. On 690 test samples, the two-stage pipeline reduced false positives from 17 to zero. In production, the system detected the node-ipc supply chain attack within 5 minutes of publication and delivered the verdict to Datadog customers within 50 minutes.

How the Math Works

The accuracy jump came from a two-stage pipeline design:

Stage 1 — Filter: An inexpensive LLM asks a single question: “Does anything look suspicious?” This filter handles the vast majority of packages — approximately 95% exit here with no further action. The filter uses diff-chunking for large files and runs cheaply enough that it can scan every new package and version published to npm and PyPI in near-real-time (10-minute crawl cycles via CouchDB and PyPI RSS feeds).

Stage 2 — Agentic investigation: When the filter raises flags, a higher-powered reasoning agent engages. This agent has access to tools: GitHub API for contributor history analysis, file inspection, dependency validation against osv.dev and Datadog’s SCA database, domain typosquatting detection against known variants, and obfuscation pattern recognition (base64 payloads, command substitution).

The agentic loop can chain multiple tool calls — checking a suspicious domain, then checking the account’s creation date, then inspecting the commit history — before rendering a verdict. This multi-step reasoning on flagged packages is what reduces the false positive rate that a single-pass classifier can’t achieve.

Context management for large packages: The most effective strategy Datadog tested combined a lightweight codemap (file structure overview) with an on-demand ReadFile() tool. Rather than processing entire package contents upfront, the agent builds a structural map and requests specific files only when suspicious signals warrant deeper inspection. This achieved 95.2% accuracy while keeping per-scan costs predictable.

What It Signals

BewAIre demonstrates a pattern that will become standard for high-volume security scanning: cheap fast filter + expensive thorough investigator. The 95% exit rate at the filter stage is the economic foundation that makes the architecture viable at ecosystem scale.

The detection categories — typosquatting, obfuscation, account metadata, supply chain compromise in high-value packages — map directly to the threat classes that caused real incidents in the past two years: the axios, LiteLLM, and Mistral compromises, and the Hackerbot Claw attack.

For engineering teams using Datadog’s SCA or Xray-equivalent tooling, this is the infrastructure explanation behind why early-warning supply chain alerts work faster than they used to. The system crawls continuously, not on a nightly schedule.

Source: Datadog Engineering

Content created with AI assistance and reviewed for accuracy.