AI Default Model Selection Is a Data Integrity Problem, Not a Preference
Agentic AIMathematician Adam Kucharski submitted identical datasets to Copilot and asked it to compare them. The system found meaningful differences that didn’t exist. When its own keyword count showed no differences, it ignored that result and fabricated percentages anyway.
The Experiment
Kucharski created 2,000 simulated responses, labeled them “UK,” then duplicated them exactly and labeled the copies “US.” After shuffling all 4,000 entries, he submitted them to Copilot’s Auto mode. Copilot concluded the responses “differ mainly in tone, intensity, and wording style.” The analysis was completely invented—the underlying datasets were identical.
The second experiment was more revealing. He generated 200 career goal statements and replicated the same dataset five times, labeling each version for a different country: US, UK, France, Germany, and Italy. Copilot’s response claimed that “Italians were three times more likely to show interest in arts careers than Brits” and that “Americans were 1.5 times more business-oriented than the French.” Every assertion was fabricated. All five datasets contained the same statements.
The moment that clarifies what’s happening: when prompted to validate its findings, Copilot ran a keyword count that correctly identified no differences between the datasets. Then it ignored that result and presented an alternative analysis with invented percentages.
Which Models Failed, and Which Didn’t
The Decoder’s testing with Kucharski’s career goals scenario found that Copilot Auto and Gemini Flash 3.5 both generated stereotypical responses, fabricating country-specific differences from the identical data. ChatGPT Instant and Claude Opus 4.7 automatically engaged extended reasoning, wrote Python code to analyze the data, and correctly identified that the datasets were duplicated across all five countries.
Manually switching Copilot and Gemini to their reasoning models produced accurate results. The failure isn’t inherent to the products—it’s in the default routing. Microsoft’s Auto mode is described as automatically selecting the optimal model for each task. In this experiment, it selected a fast model that pattern-matched to training data stereotypes rather than analyzing the actual input.
The relevant detail is the scale of default exposure. The majority of Copilot users operate in Auto mode without visibility into which model is running. They’re not choosing speed over accuracy—they’re accepting a routing decision they’re not aware is being made.
The So What
This isn’t primarily a story about Copilot. It’s about what AI tools do with ambiguous analytical tasks when the model can’t distinguish real differences in the data. The pattern is consistent: fast models generate plausible-sounding analysis. When the data doesn’t support a conclusion, they generate one from training priors. The presentation looks like analysis of the input. The content is inference from training data.
The practical implication for anyone using AI assistants in research or data analysis contexts is to verify that the model actually processed the data rather than pattern-matched to expectations. Kucharski’s recommendations are specific: document expected results before running AI analysis, perform sanity checks before accepting AI-generated findings, and avoid changing models after seeing results that seem to confirm a hypothesis.
The easiest structural check: ask the model to describe the raw data before asking it to draw conclusions. If the description doesn’t match the actual data, the analysis won’t either.
Photo by Sylvain Cls on Pexels
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →