AI Search Agents Don't Actually Search — They Confirm

MiniMax M2.5 solved 44.5% of web research tasks from memory alone — without internet access. When researchers removed supporting documents from the search indexes those agents were querying, performance collapsed from 44.5% to 8.0%. The agents weren’t researching the web. They were confirming what they already knew and finding supporting documents they had effectively predicted would exist.

Intrinsic Knowledge Dependence

Researchers from Harbin Institute of Technology and Xiaohongshu have named this phenomenon intrinsic knowledge dependence (IKD): the tendency for AI search agents to rely on internal training knowledge rather than conduct genuine web research. The mechanism is model-led rather than evidence-led.

As searches progress, agents generate 70–80% of subsequent queries from their own hypotheses rather than from previous findings. Models use discovered evidence fewer than one-third of the time — the paper reports actual evidence use rates of 24.7–32.2% across the tested models. The remaining queries are generated from the model’s internal reasoning loop, which means the agent is searching to confirm a narrative it is already constructing, not to discover what exists in the world.

Kimi K2.6 scored 62% on BrowseComp-ZH, the Chinese variant of the research benchmark, without using tools at all. When tools were provided and documents removed from indexes, that score dropped to 2.3%. The gap between with-tools-plus-documents and with-tools-alone reveals exactly how much of the benchmark performance was coming from training data rather than actual web discovery.

The LiveBrowseComp Problem

The deeper issue is that existing benchmarks are becoming unreliable measures of research capability. Static benchmarks like BrowseComp contain questions whose answers have migrated into model training data across successive training runs. A model that memorized the answers performs the same as a model that actually researched them — benchmark scores are indistinguishable.

To address this, researchers created LiveBrowseComp: 335 questions requiring facts from the 90 days preceding the benchmark’s creation date, forcing genuine retrieval. The results showed dramatic ranking shifts. DeepSeek v3.2 ranked at the bottom on the static BrowseComp but climbed to the top on LiveBrowseComp. The inverse was also true: models that appeared to be strong researchers on static benchmarks turned out to be strong memorizers.

For teams building agentic AI stacks that depend on web research — pricing intelligence, competitive monitoring, regulatory tracking, real-time event processing — this finding is structural rather than incidental. An agent that scores well on research benchmarks may be drawing from training data, not from the web. The distinction is invisible until something changes in the real world that hasn’t yet been reflected in training.

Implications for Stack Design

The practical consequence is that agentic research pipelines should not treat high benchmark scores as a proxy for real-time retrieval quality. Teams building production agentic workflows with web research components need evaluation frameworks that specifically test against recent, out-of-training-distribution information — not questions whose answers are effectively memorized.

Two design principles follow from the IKD finding:

First, retrieval verification: if an agent is claiming to retrieve a fact from the web, the architecture should log the actual retrieved document and verify the claim originates from that document rather than from a model-generated prediction about what would be found. This is especially critical for factual claims in regulated industries where source traceability matters.

Second, temporal freshness signals: benchmarking agents against questions with answers that changed within the past 60 to 90 days — and tracking performance on that subset separately from static benchmarks — surfaces whether a production agent is genuinely searching or primarily retrieving from internal weights.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.