LangGraph Agents with Datadog: The Bottleneck Isn't the Tools

When you instrument a LangGraph agent with Datadog, one finding consistently surprises teams: agent spans (LLM decision-making) are roughly twice as slow as tool spans (external API calls, web search, data retrieval). In traditional distributed systems, external API calls are the bottleneck. In agentic systems, the bottleneck is the model itself.

Why This Matters for Optimization

Traditional performance optimization targets the slowest external dependency. In a LangGraph agent running a ReAct-style loop — agent reasoning node, routing node, tools execution node — that instinct directs attention to the tools: Tavily web search, AWS SNS, database calls. Datadog’s instrumentation reveals that this is the wrong optimization target. The LLM decision step, not the tool execution step, consumes the bulk of latency.

This inverts the optimization strategy. Improving agent performance means reducing LLM call count, improving prompt efficiency, and managing token consumption — not optimizing tool integration speed. A faster search API won’t meaningfully reduce total session time if the model’s reasoning step dominates.

Pre-requisites

• LangGraph agent (Python)
• Datadog account with APM enabled
• ddtrace Python library
• Environment variables: ML_APP_NAME, DD_API_KEY, DD_ENV, DD_VERSION, DD_HOSTNAME
• LLM access via Amazon Bedrock (Claude Sonnet 3.5 in the reference architecture) or direct Anthropic API

Steps

1. Install and configure ddtrace

pip install ddtrace

Add to your agent entry point (5 lines of instrumentation):

from ddtrace.llmobs import LLMObs
LLMObs.enable(
    ml_app=os.environ["ML_APP_NAME"],
    api_key=os.environ["DD_API_KEY"],
    agentless_enabled=True,
)

2. Build the ReAct agent graph

The reference architecture uses a three-node graph: Agent node (LLM reasoning), Route node (conditional dispatch), Tools node (execution). LangGraph’s StateGraph connects them with conditional edges:

from langgraph.graph import StateGraph
graph = StateGraph(AgentState)
graph.add_node("agent", agent_node)
graph.add_node("tools", tool_node)
graph.add_conditional_edges("agent", route_function, {"tools": "tools", END: END})
graph.add_edge("tools", "agent")

3. Instrument tool calls with @tool decorator

Each tool function decorated with @tool is automatically traced by ddtrace. Tavily search and SNS publishing both appear as discrete spans in the trace waterfall, enabling isolation of which specific tool call caused slow responses or failures.

4. Enable cost tracking

Datadog APM captures token-level cost estimates per provider. In the reference example, a full agent session cost $4.61 — broken down by input tokens, output tokens, and provider. This provides the data foundation for step-level cost attribution and Fast Mode routing decisions.

5. Configure security evaluators

from ddtrace.llmobs.evaluators import PromptInjectionEvaluator

Security evaluators run on every LLM input, catching adversarial prompts attempting guardrail violations. Alert thresholds can be configured to trigger on prompt injection detection before a response is generated.

The Trap

Token counting via python3 -c "print(len(text))" underestimates actual LLM cost — token count and character count diverge significantly for code, structured data, and multilingual content. Use Datadog’s provider-reported token counts rather than local estimates for cost attribution.

Cited References

• LangGraph Agent Monitoring — Datadog
• LangGraph Documentation
• Datadog LLMObs SDK Reference
• Datadog AI Agents in Production — prior Stack Archive coverage
• Production RAG observability patterns

Source: Datadog Blog

Content created with AI assistance and reviewed for accuracy.