AI Coding Agents Find the Right File — But Cover Only 14–19% of the Critical Lines
Agentic AIThe assumption embedded in most coding agent benchmarks is that finding the file is the hard part. A new study from Shanghai Jiao Tong University and collaborators challenges that assumption — and finds the real gap is several layers deeper.
The SWE-Explore Benchmark
SWE-Explore isolates the code-search phase of agentic repair from the final fix. Rather than measuring whether an agent produces a working patch, it measures whether the agent found the right code first — before any repair attempt begins.
Ground truth is derived from successful repairs themselves: if multiple advanced models independently converge on the same code sections when fixing a bug, those sections are considered genuinely necessary. The dataset spans 848 problems across 203 open-source repositories in ten programming languages.
Models tested include Claude Code, GPT series, Gemini, OpenHands, Kimi K2 variants, and specialized systems including CoSIL, AutoCodeRover, and OrcaLoca.
The Finding
Agents navigate to the correct source files at rates that look promising. But the granularity gap opens at the line level. General coding agents cover 14–19% of the lines that are actually needed to repair a bug — even when correctly positioned in the right file.
The benchmark reveals a clear threshold: repairs consistently fail when agents cover less than 50% of the essential context. The current generation of agents is structurally underexploring the code they locate.
The Counterintuitive Implication
The design pattern of most production coding agents is to filter aggressively — skip what looks irrelevant, minimize tokens consumed. SWE-Explore’s results suggest this design choice is the primary driver of low line-level coverage.
Systems that read more code — broader exploration rather than targeted precision — outperform those optimized for efficiency. “Filter less, read more” is the structural recommendation, not a style preference.
The framing matters for anyone deploying these agents in production incident response or automated bug triage. The bottleneck is not the model’s reasoning quality on code it reads. It is how much of the relevant code the agent chooses to examine before generating a response.
So what: Benchmarks that measure only final repair success mask this gap entirely — an agent can produce a plausible-looking patch from partial context and still pass end-to-end tests. If your team relies on coding agents for root-cause analysis or autonomous bug fixes, the architectural question is not which model to use. It is whether your agent’s exploration budget is set wide enough to capture the code that actually governs the behavior you’re trying to fix.
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →