GPT-5.5: The Benchmark Leader With an 86% Hallucination Rate
Agentic AIGPT-5.5 now leads the Artificial Analysis Intelligence Index with 60 points — three ahead of Claude Opus 4.7, Gemini 3.1 Pro Preview, and GPT-5.4, which are all tied at 57. It also posts an 86% hallucination rate on the AA-Omniscience benchmark, compared to Claude Opus 4.7’s 36%, Gemini 3.1’s 50%, and Grok 4.20’s 17%. The model that leads the capability rankings is simultaneously the second-least reliable on factual recall.
Context
AI benchmark leadership and production reliability have always diverged to some degree. Controlled benchmarks test specific capabilities on curated tasks; production workloads introduce distribution shift, adversarial inputs, and failure modes that controlled tests don’t surface. What makes GPT-5.5’s positioning unusual is the magnitude and direction of the divergence.
The conventional expectation is that models improve across dimensions simultaneously — higher benchmark scores correlate, at least loosely, with lower hallucination rates. GPT-5.5 breaks that correlation. Its accuracy on factual knowledge questions is the highest of any tested model at 57%, and its hallucination rate is the highest of any tested model at 86%. It knows more than its competitors and asserts incorrect information more frequently.
The Detail
The Artificial Analysis Intelligence Index, which aggregates performance across reasoning, coding, and knowledge benchmarks, puts GPT-5.5 three points clear of the field. The AA-Omniscience benchmark, which tests factual accuracy and hallucination resistance, tells a different story: GPT-5.5 at 86% hallucination is more than twice as likely to confidently assert incorrect information as Claude Opus 4.7 at 36%, and five times as likely as Grok 4.20 at 17%.
The pricing structure adds another dimension. GPT-5.5 is priced at $5 per million input tokens and $30 per million output tokens — double GPT-5.4’s $2.50/$6. The net API cost increase is approximately 20% because GPT-5.5 uses roughly 40% fewer tokens to complete comparable tasks. The efficiency gain reduces but does not eliminate the price premium relative to the previous generation.
BullshitBench, which measures model resistance to false premises and leading questions, shows a related pattern: GPT-5.5 pushes back on incorrect premises about 45% of the time, while GPT-5.5 Pro drops to 35%. Claude models lead the overall BullshitBench leaderboard. A model that scores 86% on hallucination while maintaining 57% factual accuracy is one that is both capable and overconfident — it retrieves correct information at a high rate while also generating incorrect assertions at a high rate, without clearly distinguishing between them.
This is the dangerous asymmetry for production deployment. The peer review AI hallucination research documented the same pattern in domain-specific contexts: models that perform well on structured tasks can simultaneously fabricate with confidence in adjacent domains.
The So What
For teams evaluating models for production agentic systems, GPT-5.5’s benchmark leadership and hallucination profile point to a use-case-specific selection question rather than a global capability ranking. Tasks with well-defined evaluation criteria, available ground truth, or human-in-the-loop review loops can absorb higher hallucination rates because errors are catchable. Tasks that generate outputs consumed without verification — financial analysis, research summarization, client-facing documents — are exactly where an 86% hallucination rate matters regardless of benchmark position.
The benchmark-reliability gap is becoming a more relevant evaluation dimension than benchmark ranking alone. For any production deployment decision, hallucination rate on the target task distribution is the number that matters — not the composite index score.
Photo by Google DeepMind on Pexels
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →