CEO-Bench: Three AI Models Survived 500 Days Running a Startup

Princeton researchers handed 14 AI models the keys to a fictional software startup and let them run it for 500 simulated days. Three came out ahead of the $1 million they started with. A simple rule-based heuristic — no language model, no reasoning — beat eleven of them.

The Number

CEO-Bench is a longitudinal benchmark from Princeton researchers. Agents managed NovaMind — a fictional software company — with access to 34 tools and 19 database tables, starting with $1 million in cash and zero customers. Bankruptcy triggered if cash hit zero. The three models that survived 500 days in profit:

• Claude Fable 5: $47.15 million — the only model to finish profitable across multiple independent runs
• Claude Opus 4.8: $27.8 million
• GPT-5.5: $21.3 million

The rule-based heuristic baseline — a simple, non-adaptive decision framework with no LLM — reached $15.76 million. Eleven frontier AI models underperformed it.

How the Math Works

The winners shared a specific trait: sustained strategic coherence. Fable 5 explored and adapted its approach throughout the 500-day window. Opus 4.8 built internal simulations to project outcomes before committing to actions. GPT-5.5 tracked and analyzed negotiation history to improve deal outcomes on repeat interactions.

The losers followed a pattern the researchers describe as failing “to maintain coherent strategy over time.” Opus 4.7 — one generation behind Fable 5 in Anthropic’s own lineup — adopted passive cost-cutting: the AI equivalent of laying off staff and waiting. Most models plateaued early, reaching a local optimum and refusing to adapt. The theoretical maximum achievable in the benchmark is approximately $2.2 billion, which puts even the top finisher at about 2% of optimal.

What It Signals

CEO-Bench measures something one-shot evaluations miss: stamina. Not just whether a model can reason well, but whether it can stay goal-directed across the equivalent of 18 months of business decisions without losing the thread.

The heuristic result is the finding that deserves the most attention. A rigid, non-adaptive rule framework outperformed 11 frontier LLMs on a task that rewards adaptive strategy. That gap doesn’t argue against using LLMs — it argues for layering them over structured decision scaffolding rather than running them unconstrained. For teams evaluating AI agents for long-horizon automation, the benchmark suggests hybrid architectures: LLM reasoning for judgment calls, explicit strategy frameworks for temporal coherence.

See also the gap between AI benchmark performance and real-world labor outcomes — a pattern that extends beyond CEO-Bench.

Source: The Decoder

Content created with AI assistance and reviewed for accuracy.