Model Routing Promises 85% Savings. One Switch Can Erase All of It.

Model routing has become the default cost-optimization move for AI agents: send routine work to a cheap model, save the expensive one for hard problems. JFrog’s research shows the same technique can also be the fastest way to blow your inference budget.

Why Everyone Believes This

The logic is sound on paper. Route routine tasks to a cheaper model like Claude Haiku and reserve frontier models like Claude Sonnet for complex reasoning, and — done correctly — routing can cut inference costs by 40-85%. That’s a big enough number that “add a router” has become close to a default recommendation for teams trying to control agentic AI spend, and it’s easy to assume routing is a strict win: cheaper model in, cheaper bill out, no downside.

What the Evidence Shows

The catch JFrog identifies isn’t in the routing logic itself — it’s in prompt caching, the other major cost-control mechanism most agentic sessions rely on. Every provider’s prompt cache is model-specific. Switch models mid-conversation and the new model starts with a cold cache, reprocessing the entire conversation history at full price, “as if Turn 1 never happened.” For a 20,000-token session, JFrog states plainly that a single mid-session model switch can cost more than the savings the switch was meant to generate — the exact opposite of the intended outcome. The failure isn’t theoretical: it shows up in specific, common patterns — switching models mid-conversation without summarizing first, enabling auto-routing in an LLM gateway partway through a session, modifying tools or system prompts mid-session, or running compression calls on a separate model from the main session.

The Correction

Routing still works — the fix is in when and how the switch happens, not whether to route at all. JFrog’s recommendation is to keep sessions short and delegate complex work to subagents early, before a long context history accumulates, rather than switching the primary model mid-conversation. Each subagent then runs on its own optimal model with only the context it actually needs, sidestepping the cache-invalidation problem entirely because there’s no mid-session switch to trigger it. Where a switch is genuinely unavoidable, summarize deliberately first rather than carrying the full raw history into the new model’s cold cache, and lock session configuration — system prompt, tools, and model choice — at the start rather than modifying it live. JFrog positions its own CLI tool, Boost, as a complementary fix on the context side: it compacts terminal output during agentic loops, stripping repetitive boilerplate while preserving exact error traces, which reduces the context bloat that makes cache invalidation so expensive when it does happen.

Source: JFrog Blog

Content created with AI assistance and reviewed for accuracy.