GPT-6 Astra's ARC-AGI-3 Score: What the 37-Point Benchmark Gap Actually Means
Agentic AIGPT-6 Astra scored 99.9% on ARC-AGI-3 under OpenAI’s proprietary harness. It scored 62.7% under the standard evaluation harness. Both numbers are real. Neither tells the complete story.
What Happened
OpenAI launched GPT-6 Astra this week. Benchmarks followed immediately — and reached contradictory verdicts. Epoch AI, running 50+ evaluations, ranked Astra first overall across its Intelligence Index. Artificial Analysis rated it equivalent to its predecessor GPT-5.6 Sol and behind Claude Fable 5.1 on their composite metric.
The central disagreement is the ARC-AGI-3 harness controversy. The Decoder’s analysis documents the gap: on the standard ARC Prize evaluation harness, Astra scored 62.7% at a cost of approximately $26,000. On OpenAI’s proprietary harness, it scored 99.9%. That 37-percentage-point spread on identical tasks, from identical underlying capability, reflects methodology — not the model.
Why It Matters
The ARC Prize benchmark, designed by François Chollet, was built specifically to resist memorization and test genuine reasoning. It has served as one of the few credible independent measures of AI progress. The harness standardization was supposed to make scores comparable across vendors.
OpenAI’s proprietary harness changes that. The exact mechanics are not fully disclosed, but the ARC Prize organization has been explicit: only the standard-harness score (62.7%) enables fair vendor comparison. The 99.9% figure is real performance under optimal conditions — but those conditions are OpenAI’s own, and they cannot be replicated by competitors or auditors.
The broader implication is structural. If vendors can submit scores under proprietary evaluation conditions, the benchmarking ecosystem loses its function as an independent check. Epoch AI and Artificial Analysis reaching opposite conclusions on the same model is the symptom; the cause is that “AI performance” is increasingly measurement-method-dependent.
What This Means for Builders
Chollet has updated his AGI timeline. His prior forecast was approximately 2030 for ARC-AGI-3 saturation. He now states that Astra arrived “approximately twice as fast” as anticipated, and has moved his estimate forward. On FrontierMath Erdős — a benchmark testing novel mathematical conjecture-solving — Astra is the only model to have solved open problems at all, scoring 3%.
For practitioners evaluating models for agentic workloads, the takeaway is practical: benchmark scores are increasingly vendor-specific claims that require independent replication before they inform architecture decisions. A 62.7% standard-harness score on ARC-AGI-3 still represents a significant capability jump from GPT-5.6 Sol’s 7.78% on the same benchmark. The headline 99.9% is a claim about what Astra can do in optimal conditions — which may or may not match your deployment environment.
The So What
GPT-6 Astra is a meaningful capability advance. The disagreement between benchmark providers is not noise — it is signal about the state of AI evaluation infrastructure. As models approach human-competitive performance on reasoning tasks, the measurement systems designed to track progress are struggling to keep pace with vendor incentives. Independent evaluation harness adherence matters more now, not less, and practitioners should weight standardized scores over proprietary ones when comparing models.
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →