The Last Six Months in LLMs: What Actually Shifted

Simon Willison’s six-month recap of the LLM landscape identifies three changes that matter beyond the benchmark scoreboard: model leadership rotated five times in six months with no durable winner, coding agents crossed from “often work” to “mostly work” in November 2025, and open-weight models on consumer hardware are performing at levels that would have been frontier-quality a year earlier.

The Benchmark Treadmill

Model leadership at the top of standard benchmarks changed hands five times between November 2025 and May 2026 — Claude 4.5 → GPT-5.1 → Gemini 3 → GPT-5.1 Codex Max → Claude Opus 4.5 — with each cycle measured in weeks, not months. The implication Willison draws is not that the frontier is advancing rapidly (which it obviously is), but that any specific model capability advantage is ephemeral at this cadence. Product strategies built on “we use the best model” are product strategies built on a constantly moving benchmark.

The coding agent transition in November 2025 is the more operationally significant data point. Willison marks this as the moment when agentic coding systems shifted from a “will it work?” question to an “how do I deploy it?” question for most professional programming tasks. The shift in the probability distribution — from “often work” to “mostly work” — is the threshold that changes whether agents are worth integrating into production workflows.

The Open-Weight Surprise

The open-weight model results are the counterintuitive finding. Qwen3.6-35B-A3B, at 20.9GB, outperformed Claude Opus 4.7 on certain benchmarks. Gemma 4 leads among U.S. open-weight models. GLM-5.1 from China runs 754 billion parameters as open-weight. These are models that run on consumer hardware, not data center clusters.

The “Claw” phenomenon is the most tangible expression of this: demand for Mac Minis as local agent platforms became large enough to create visible supply shortages in Silicon Valley. Developers running always-on local agents at home, on hardware bought specifically for that purpose, generating demand spikes measurable in retail inventory. That’s a behavioral signal, not just a benchmark result.

The open-weight capability improvement doesn’t eliminate the case for frontier API access — Opus 4.8’s 69.2% on SWE-Bench Pro, parallel sub-agent workflows, and Fast Mode pricing all remain compelling for professional agentic applications. But it changes the economics of the low-latency, high-privacy, edge-inference tier. Tasks that required frontier API access for acceptable quality a year ago now have viable local alternatives that don’t leave your hardware.

Source: Simon Willison

Content created with AI assistance and reviewed for accuracy.