BankerToolBench: 500 Bankers, Zero AI Outputs Ready for Clients
Agentic AINot a single AI-generated investment banking output — across GPT-5.4, Claude Opus 4.5 and 4.6, Gemini 3.1, Grok 4, and five other frontier models — was rated ready to send to a client without edits. 27% of outputs were rated completely unusable. The benchmark, BankerToolBench, was designed by 502 investment bankers from Goldman Sachs, JPMorgan, Evercore, Morgan Stanley, and Lazard — professionals who log 5-hour tasks as routine and 21-hour tasks as expected.
Context
AI performance on financial reasoning benchmarks has improved substantially. Models that could not competently handle DCF models two years ago now score reasonably on structured financial tasks, and frontier models like GPT-5.4 demonstrated credible Excel reasoning capabilities earlier this year. BankerToolBench tests a different question: not whether AI can reason about financial concepts, but whether the outputs it produces meet the professional standards required for client delivery.
Those standards are unusually high. Investment banking deliverables must be internally consistent across multiple files, correct on formulas that update dynamically with changing assumptions, and presentable at a level of polish that reflects the fees charged. The gap between “technically correct” and “client-ready” is wider in investment banking than in almost any other professional context.
The Detail
BankerToolBench was developed by Handshake AI and McGill University with 502 investment banking professionals. The 172 bankers who designed the benchmark tasks logged over 5,700 hours creating evaluation criteria, producing rubrics that average 150 criteria per task. Tasks mirror real junior-banker work: Excel financial models, PowerPoint pitch decks, PDF reports, and Word memos — each averaging 5 hours of human completion time, with some requiring up to 21 hours.
The results across nine frontier models were consistent in their failure mode: 0% of outputs were rated ready to send as-is, 13% needed light edits, 19% required moderate rework, 41% needed major overhauls, and 27% were completely unusable. The best-performing model, GPT-5.4, achieved a 16% pass rate when required to produce three consistent outputs in a row — still far from production-ready for direct client delivery.
The model-specific finding that reveals the most about failure patterns is Claude Opus 4.6’s Excel output. The deliverables appeared polished visually, but a structural examination showed that key numbers were hardcoded as fixed values rather than calculated through formulas. In investment banking, this is a categorical failure: changing the purchase price in a merger model should update every dependent cell automatically. A model that hardcodes values cannot support scenario analysis — the fundamental analytical workflow in investment banking.
The hardest task categories were debt capital markets, merger models, and capital structure tables. These are the deliverables where formula integrity, cross-file consistency, and financial logic are most precisely specified, and where hallucinated numbers are most consequential.
The findings align with a broader pattern in multi-agent financial analysis research: models can navigate financial reasoning tasks at a conceptual level while producing outputs that fail professional standards on structural details.
The So What
BankerToolBench is a useful calibration for how to interpret frontier model capability claims in high-stakes professional contexts. Benchmark performance on reasoning or coding tasks does not transfer directly to domain-specific professional deliverables. The requirements for investment banking work — formula integrity, dynamic modeling, multi-file consistency, professional polish — are not captured in general-purpose benchmarks.
For engineering and platform teams building AI systems for finance workloads, the benchmark results suggest that output review pipelines are not optional overhead. 69% of bankers said they’d use AI outputs as a starting point — which means value exists, but only with professional review built into the workflow. The 0% pass rate for direct delivery is not a ceiling on AI utility in finance; it is a measurement of where human-in-the-loop supervision remains non-negotiable.
Photo by Hanna Pad on Pexels
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →