How AWS Engineers Trace Agentic AI Failures with OpenTelemetry and OpenSearch
Cloud Platform EngineeringAgentic AI failures look like application failures from the outside and like reasoning failures from the inside — and most observability stacks weren’t built to handle the difference. AWS has published a working pattern that changes this.
The Observability Gap for Agent Systems
Classical observability covers logs, metrics, and traces for deterministic systems. An agent workflow introduces a third failure mode: the model calling the wrong tool, in the wrong order, or with a malformed context that only exposes itself three steps later in the reasoning chain. Standard application tracing captures that something failed. It doesn’t show why — which turn in the agent’s reasoning produced the error, or whether the failure was in the tool invocation, the context retrieval, or the model’s interpretation of the intermediate state.
OpenTelemetry provides the instrumentation layer that makes agent reasoning chains traceable. Each step in an agent’s workflow emits spans — named units of work with timing and attribute data — that can be assembled into a causal trace of the full reasoning sequence. The missing piece has been a purpose-built analysis surface for those agent-specific traces.
What AWS Built
The pattern centers on OpenSearch Agent Health, an open-source evaluation and observability framework added to Amazon OpenSearch Service. Agent Health brings three capabilities together in a single interface: application monitoring, Prometheus metrics via PromQL queries, and AI agent trace analysis. Engineers can query standard infrastructure metrics alongside agent-specific spans in the same OpenSearch UI session, eliminating the context switch between an APM tool and a separate agent debugging surface.
The operational outcome described in AWS’s walkthrough is tracing an AI agent’s full reasoning chain down to the specific failing tool call — and drilling from a service-level health view to the exact span that caused a downstream failure. The pattern is designed to be portable: the OpenTelemetry instrumentation can run against any agent framework, and the OpenSearch analysis layer is open-source, which means the approach isn’t locked to AWS-managed agents.
What the pattern specifically targets: wrong tool calls (where an agent selects an incorrect function for a task), latency spirals (where a multi-step workflow balloons in cost and time due to redundant retrieval), and context cache expiration penalties (where cached context expires mid-workflow and forces expensive re-inference). Each of these failure modes is invisible to standard application tracing but produces a distinctive signature in agent-level span data.
The So What
Platform engineers deploying agentic AI in production now have a reference architecture for making agent failures inspectable at the same granularity as application failures. The practical engineering leverage is in catching these failures before customers encounter them: running OTel-instrumented agent workflows in staging, then using OpenSearch Agent Health to identify which reasoning steps consistently fail or produce latency outliers, before promoting to production. The deeper implication is that “observability” for AI systems means something different than it did for microservices — the trace must capture the model’s decision path, not just the infrastructure response time. AWS has operationalized that distinction with tooling that’s available today. Related: how cloud-native runtime verification is evolving for agentic workloads.
Source: The New Stack
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →