Rippling Tested 15 AI Models on Real Payroll Data. The Cheapest Tied the Most Expensive.
EntrepreneurshipThe AI model procurement decision at most companies is based on benchmark leaderboards, vibes from developer teams, and the assumption that higher cost correlates with better output. Rippling just ran the experiment that most companies won’t: 2,100 scored agent runs per model, on actual production payroll tasks, across 15 models. The finding is uncomfortable for anyone who has been paying premium rates.
The Number
Z.ai’s GLM 5.2 cost 85% less than the most expensive models tested and produced nearly comparable results on Rippling’s real payroll data. The methodology wasn’t a theoretical benchmark: Rippling’s President and CPO Matt MacInnis published the results of scoring 2,100 agent runs per model on tasks that reflect actual work inside Rippling’s payroll system — headcount queries by department and tenure, salary adjustments for employees meeting specific conditions, new hire onboarding flows, termination routing with approval chains, and manual spreadsheet data entry into pay runs.
These are not cherry-picked demonstration tasks. They are the operational workload Rippling’s AI infrastructure handles.
How the Math Works
The implications extend beyond the per-model ranking. Rippling reduced its total token spending from 40% of its R&D headcount budget in early 2026 to 15% by July — primarily by identifying which models delivered equivalent results at lower cost, then shifting workloads accordingly. July’s token bill equaled 37% of April’s.
That 63% cost reduction was not achieved by reducing AI usage. It was achieved by using the right model for each workload category. The company also built an internal AI Spend Console to make this optimization visible at the employee level — surfacing which team members were generating AI spend and what their productivity output looked like in return, alongside a rework rate metric that tracks AI-generated work that required human correction.
What It Signals
The Rippling test does something most AI model discussions don’t: it measures performance on domain-specific production tasks rather than general benchmarks. The finding that GLM 5.2 competes with premium models on payroll operations is not a claim that GLM 5.2 is generally superior — it is a claim that on these specific tasks, the cost differential isn’t earning anything.
For SaaS companies and operators building AI-native workflows, this argues for a straightforward process: define your workload categories, build a scoring methodology against real tasks in those categories, and run the models against your data before committing to a cost structure. The leaderboard is a starting point for exploration, not a procurement guide.
The pattern of AI agents displacing SaaS tools is accelerating — but the model selection decisions being made to power those agents are frequently not grounded in workload-specific evidence. Rippling’s methodology is reproducible, and the ROI on running it is clear from their own numbers.
The So What
If your AI infrastructure cost is based on “we used the model with the best benchmark scores,” you are probably overpaying. The relevant question isn’t which model scores highest on MMLU — it’s which model scores best on your tasks, at your data volumes, in your operational context. Rippling’s 85% cost differential on real payroll work is not an edge case; it’s what domain-specific model selection looks like in practice. Run your workload against at least three models at different price points before locking in a cost structure. The savings are there; they just require the experiment.
Source: SaaStrAI
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →