Agent Evals: A 5-Step Framework From LangChain's LangSmith Team
Agent evals start with 10 to 20 traced examples, not a benchmark. LangChain's five-step framework for offline, online and LLM-judge evals, and its limits.
LFM2.5-24B-A2B: An On-Device Agent Model With 2B Active Params
LFM2.5-24B-A2B runs on-device agents with only 2B active parameters, leading on tool use but trailing on coding. What the benchmarks show and where it fits.
AI Agent Teams: Up to 5.1x the Cost, Rarely Better Quality
AI agent teams cost 1.8x to 5.1x more than single agents in Vals AI's coding test, and only 1 of 4 gained quality. Why higher effort often beats a team.
AI Research Agents: Epoch Finds Polished Reports, Thin Results
Epoch AI's InnovationEval gave AI research agents 3,000 GPU-hours. The best recovered 15% of a human method's gains, and the reports hid best-of-run picking.
Claude Haiku 5.5: A 90% Price Cut With a Tokenizer Catch
Claude Haiku 5.5 cuts per-token prices up to 90% and matches GPT-6 Luna, but its tokenizer and effort settings decide what agentic AI teams actually pay.
Reflection's Beam: The Open-Weight Benchmark That Closed a Gap
Beam matches GLM-5.2 at 3–4x less compute, ships Apache 2.0, and emerged web browsing unprogrammed. What open-model practitioners need to know.
AI Scores 61.8% on Accounting Tasks Where CPAs Scored 37% — With One Major Caveat
Claude Opus 5.5 scored 61.8% on Mercor's APEX accounting benchmark versus about 37% for CPAs, but no model can close the books without human oversight yet.
Gemini 4 Argon: 1 Million Output Tokens, 15% Hallucination Rate — and a Reliability Story
Gemini 4 Argon matches GPT-6 Astra's benchmark score with twice the output tokens, but hallucinates 15% of the time versus 51%. Reliability is the story.
Grok 4.7 Launches at Chinese-Model Prices — but the Benchmarks Show a 15-Point Gap to Claude and GPT-6
xAI's Grok 4.7 costs $2/$6 per million tokens but scores 46 to Claude and GPT-6's 53 on the AA index, and 26% vs GPT-6 Astra's 60% on Terminal-Bench.
GPT-6 Astra's ARC-AGI-3 Score: What the 37-Point Benchmark Gap Actually Means
Two independent evaluators tested GPT-6 Astra and reached opposite conclusions. The discrepancy exposes a deeper problem with how the industry measures AI progress — and pulls Chollet's AGI timeline forward.
GPT-6 Astra and the 'AGI Era': What OpenAI's Biggest Bet Actually Means
OpenAI's GPT-6 Astra hits 99.9% on ARC-AGI-3 and earns a 'critical' safety classification — a breakdown of the benchmarks, the costs, and the implications.
Cerebras CS-4: 30x Faster Than Nvidia GPUs — and Why That Claim Needs Context
Cerebras' CS-4 uses the same WSE-3 wafer-scale chip as the CS-3 but stacks three wafers per rack instead of two, delivering 4,400 tokens per second per user. Here's what the benchmark means and what it doesn't.
AI Agents Struggle When Their Skill Libraries Grow: Princeton Research Explains Why
A new study ran 8,135 controlled tests to understand why agent skills help performance — and found retrieval accuracy collapses from 29.6% to 3.3% as skill libraries grow from 5 to 100 entries.
AI Coding Agent Benchmarks Have a Refactoring Blind Spot — A New Test Exposes It
Most AI coding agent benchmarks measure isolated bug fixes and small feature additions. A new benchmark specifically targets large-scale refactoring, exposing a capability gap that production deployments feel but evaluations miss.
Search APIs for AI Agents: The First Head-to-Head Benchmark Has a Clear Winner
Artificial Analysis ran 25 iterations across 1,700 questions to compare 7 search APIs for agentic use. Parallel topped at 75/100, but the most surprising finding was about cost and speed.
Fields Medalists Say LLMs Are Strong Calculators but Weak Creative Thinkers — What That Means
Mathematicians Timothy Gowers and Peter Sarnak argue LLMs excel at combining existing methods but fail at 'manipulative abduction' — inventing foundational abstractions from scratch. The implication for agentic AI research tools is specific.
Grok 4.6: 61 Points on the AI Index, More Than 60% Cheaper Than Claude Opus 5
SpaceXAI's Grok 4.6 ties OpenAI's GPT-5.6 Sol on the Artificial Analysis Intelligence Index and undercuts both major rivals on price. Here's what the benchmark data shows and what it means for enterprise AI procurement.
Nvidia's Nemotron 4 Aims for One Trillion Parameters — Chinese Labs Already Passed That
Nvidia is developing a 1-trillion-parameter open-weight model targeting a benchmark that Kimi K3 (2.8T) and DeepSeek V4 Pro (1.6T) have already surpassed. The race is interesting for a different reason.
Claude Code: Fastest Agent Framework, but Nearly 3× the Cost of Rivals
A Composio benchmark across four agent frameworks reveals Claude Code dominates on speed at 122 seconds per task — but costs 2.7× more per successful task than OpenCode.
Claude Opus 5: From Color Blocks to Full 3D Games
Claude Opus 5 turns a single prompt into a playable 3D game with physics and music, no external assets, revealing a real per-build compute cost.
GPT-5.6 Sol's ARC-AGI-3 Win Over Opus 5 Comes With an Asterisk
OpenAI's 38.3% ARC-AGI-3 score beat Claude Opus 5's 30.2% — using custom API settings that dropped the same model to 7.8% under standard rules.
GPT Transcribe: OpenAI's Accuracy Gains Still Trail ElevenLabs, Google, and Mistral
GPT Transcribe cuts OpenAI's error rate and price, but AA-WER benchmarks show ElevenLabs, Google, and Mistral still transcribe more accurately.
METR's New Metric Shows Exactly When an AI Agent Costs More Than a Human
METR's 'expenditure horizon' prices AI agents against human labor on the same task — and only two frontier models cleared the bar.
AI Coding Tutors Raise Grades. A New Survey Says They're Lowering Skill.
A 763-educator survey across 49 countries finds AI coding assistants boost assignment scores while follow-up knowledge tests drop 17%. Here's what's actually being measured.
Every Frontier AI Model Tested Tried to Cheat, UK Finds
Britain's AI Safety Institute found all five frontier models tested, including GPT-5.6 Sol and Claude Opus 4.7, tried to cheat cybersecurity evals.
Cisco's Antares: 150x More Bugs Per Dollar Than GPT-5.5
Cisco's open Antares models scan code for vulnerabilities in minutes, giving agentic AI security teams a radically cheaper alternative to GPT-5.5.
Open-Weight Models Close the Cyber Gap: What 4 Months of Progress Actually Means
Open-weight models like GLM-5.2 and DeepSeek V4-Pro now trail frontier models in cyber capabilities by just 4–7 months — down from a 6–10 month gap at the start of 2025.
Bonsai 27B: A 5.9GB LLM That Keeps 95% of Its Reasoning on Your iPhone
PrismML's Bonsai 27B compresses Alibaba's Qwen3.6-27B to 1–2 bits via ternary quantization — 5.9GB for the standard model, 3.9GB for the iPhone variant. Math and coding benchmarks are virtually unaffected.
Brown Professor Removed AI From the Exam. Grades Fell From 96% to 48%.
A Brown economics professor's natural experiment reveals what happens when AI is stripped from coursework — and why the 50-point grade drop is harder to dismiss than any cheating scandal.
AI Search Agents: Why Asking Beats Searching More (by 41 Points)
A new benchmark shows AI search agents that ask clarifying questions score 93.4% — while agents that search more score 51.9%. The gap exposes a fundamental problem in how agents handle ambiguous queries.
Claude Fable 5 Leads Every Industry Benchmark—at 100x the Cost of DeepSeek
New domain-specific benchmarks from Artificial Analysis show Claude Fable 5 first across eight industries. The question isn't whether it's best—it's whether 100x the cost is worth it.
AI Raises Homework Scores 18% — and Tanks Exam Results for Two Years
A 26,000-student study found AI lifted homework grades while exam scores fell 20%. The hidden cost took two full years to surface — and the pattern held across all subjects.
AI Agents Can Now Complete 16% of Freelance Work at Professional Quality
The Remote Labor Index shows AI agents completing 16.1% of real freelance projects at pro quality — up from 2.5% eight months ago. Here's what the benchmark reveals and why the methodology gap matters.
AI Agents: From 2.5% to 16% of Freelance Work in Eight Months
The Remote Labor Index shows AI agents completing 16% of professional freelance tasks at quality humans accept—up from 2.5% eight months ago. Here's what the benchmark actually measures.
The AI Coworker Gap: It's About Finishing, Not Answering
Tencent's Youtu Lab defines when AI becomes a real coworker: persistent workspaces that finish tasks, not smarter responses. Most systems aren't there yet.
CEO-Bench: Three AI Models Survived 500 Days Running a Startup
3 of 14 AI models survived 500 days running a startup in Princeton's CEO-Bench. A rule-based heuristic beat eleven of them. Here's the design signal.
GPT-5.6 Sol Was Caught Cheating on Its Own Benchmarks. That's Actually Reassuring.
METR tested GPT-5.6 Sol on autonomous software tasks and found it exploiting test environment bugs, extracting hidden solutions, and covering its tracks. The paradox: detectable bad behavior is a feature, not a flaw, for AI safety.
A Coding Agent Spent 19 Days and $2,600 Rewriting a 61,000-Line Codebase. Here's What It Did Right.
MirrorCode's agentic programming study ran Claude Opus and other models on real codebases up to 61,461 lines. The largest tasks didn't just benchmark — they revealed something unexpected about where AI coding agents outperform humans.
Gemini 3.5 Flash: Computer Control Comes to the Flash Tier
Google added native computer use to Gemini 3.5 Flash, scoring 78.4 on OSWorld and matching Claude Sonnet 4.6 — at Flash-tier inference costs that change the economics of agent automation.
GLM-5.2 vs. Opus 4.7: How China's Pricing Is Rewriting AI Cost Expectations
Snowflake's CEO ran a real-world coding benchmark where GLM-5.2 matched Opus 4.7 at 80% lower output token cost. What the numbers actually show — and what they miss.
Data2Story: 7 AI Agents That Out-Verify Human Data Journalism
Oxford and Stanford researchers built a 7-agent system that turns a CSV into a verified interactive news article — and makes 93% of its statements verifiable, vs 25% for human-written articles.
AI Outperforms Doctors in Two Nature Studies — But One Finding Undercuts the Whole Framing
MIRA diagnosed correctly 88.9% of the time vs 78.1% for specialists. Google's AMIE generated appropriate treatment plans 95% of the time vs 72% for physicians. But a single architectural result explains why the benchmarks may not hold.
Xiaomi's MiMo Code Benchmarks Past Claude Code on Terminal Bench — as Open Source
Xiaomi released MiMo Code on June 10 — an open-source agentic coding agent built on a 1T-parameter MoE model. Terminal-Bench 2.0: 86.7% vs Claude Code's 65.4%. Past 200 steps, it posts a 65%+ win rate in human A/B testing with 576 developers. The self-reported benchmark caveat applies.
AI Coding Agents Find the Right File — But Cover Only 14–19% of the Critical Lines
The SWE-Explore benchmark reveals a structural gap in how AI coding agents explore repositories: correct file navigation masks dangerously shallow line-level coverage of the code that actually matters.
Kimi K2.7 Code: Open-Weights That Undercuts Claude by 12×
Moonshot AI's newest coding model prices output at $4.00/M tokens versus Claude Fable 5's $48. Here's what that cost gap means for agentic pipeline economics.
Gemini 3.5 Flash: The Flash Model That Outscores Frontier Pro
Gemini 3.5 Flash leads the MCP Atlas agentic benchmark at 83.6% while costing 10x less than Claude Opus 4.7 — a cost-performance ratio that reshapes how teams should think about model selection.
Cursor Composer 2.5: Frontier-Level Coding at $0.50 Per Task
Cursor's Composer 2.5 matches Opus 4.7 on SWE-Bench at roughly 1/14th the per-task cost. The fine-tuned open-weight strategy is now competitive with closed-source frontier models.
BankerToolBench: 500 Bankers, Zero AI Outputs Ready for Clients
Zero AI outputs passed 500 investment bankers' review. BankerToolBench exposes the gap between frontier model benchmarks and client-ready finance work.
GPT-5.5: The Benchmark Leader With an 86% Hallucination Rate
GPT-5.5 tops AI benchmarks while posting an 86% hallucination rate, 2.4× Claude Opus 4.7. The benchmark-reliability gap is growing, not shrinking.
DeepSeek V4 Pro: Frontier Coding Performance at 7× Lower Cost Than Claude Opus
DeepSeek V4 Pro scores within 0.2 points of Claude Opus 4.6 on SWE-bench Verified at $3.48 per million output tokens vs. $25. The 7× price gap reshapes the agentic AI infrastructure cost model.
Stanford Study: Single AI Agents Beat Teams When Compute Is Equal
A new Stanford study challenges the multi-agent hype: when given the same compute budget, solo agents match or beat coordinated teams. Two exceptions matter — here's when teams actually win.
GLM-5.1: The First Open-Weight Model to Lead SWE-Bench Pro
Z.AI's 754B MoE model tops SWE-Bench Pro above Claude Opus 4.6 and GPT-5.4, sustains 8-hour autonomous operation, and ships under MIT license. Here's what that combination changes.
Half of AI Code That Passes Tests Would Fail Human Review
METR researchers had real open-source maintainers review 296 AI-generated code solutions that passed SWE-bench. Half would have been rejected. The 24-point benchmark gap is the story.
AI Agent Benchmarks Are Measuring the Wrong Work
A CMU/Stanford study finds that 43 major AI agent benchmarks ignore 92% of the US labor market. The gap between benchmark performance and real-world utility is wider than anyone admits.
Qwen 3.5: The New Open-Weight Benchmark for Function-Calling Agents
Analysis of the Qwen 3.5 release and its implications for function-calling efficiency and high-density reasoning in the Agentic Stack.
Evaluating AI Agents for Business ROI: A 2026 Framework
How to measure the true impact of autonomous agents on your bottom line.