LangSmith AI Search: Finding Agent Failures by Behavior

In a sample of 150 conversations from Open SWE, LangChain’s open source coding agent, about 85% of tokens were tool output, mostly file reads and shell commands. That figure, from LangChain’s launch post for AI Search in LangSmith, explains why finding the trace where an agent misbehaved is so hard for teams running agents in production: the behavior you’re looking for sits in a few user and agent messages buried inside mostly tool output.

What Happened

LangChain released AI Search, which lets users query agent traces in natural language by what happened rather than by keywords or metadata. The post’s examples include “did the user get frustrated?”, “did PII leak?”, an agent claiming it finished a task when it didn’t, and users confused about onboarding. Matches can go straight into eval datasets or annotation queues.

AI Search is in public beta on LangSmith Cloud in the GCP US, EU, and APAC regions, “at no additional cost” during the beta. Org admins enable it in settings, and it works alongside existing filters such as errored runs or tags, per the LangSmith docs. It accepts traces from the OpenAI and Anthropic SDKs, LangChain, LangGraph, the Vercel AI SDK, OpenTelemetry, and others.

Why It Matters

The design choice is to skip plain vector search. LangChain argues that an embedding compresses a whole trace into one vector, so a trace where the agent succeeded and one where it only claimed success can look nearly identical. Instead, structured filters narrow the candidates, and a fast decision model reads each remaining trace in parallel and answers a narrow question about it at search time.

Traces are too long to read whole. In the Open SWE sample, some conversations exceeded 1 million tokens, and the median was about 72K tokens, of which about 2K were user messages and agent replies. So LangSmith’s SmithDB first converts each session into a deduplicated trajectory, then fits it into a fixed token budget (the post illustrates 8K) by priority: user messages and agent replies first, then intermediate reasoning and tool calls, then tool results, then system messages.

What This Means for Builders

The budget is also the limit. The authors say it works when the answer lies in what the user and agent said. A question about whether an agent ignored a tool error needs the tool results, which are the first thing cut. Subtle distinctions are hard too: “the user gave up” and “the user got what they needed and left” may turn on a couple of turns in a long session. LangChain lists question-aware context selection and a second pass on uncertain results as future work. Teams building their own trace-to-experiment loops face the same trade-off between cost and how much of a trace a model reads.

Source: LangChain Blog

Content created with AI assistance and reviewed for accuracy.