Building Production Agentic AI: What IBM Learned the Hard Way

The biggest surprise in IBM’s production agentic AI deployment wasn’t the agent framework or the model selection. It was the realization, months into development, that identifying which internal APIs existed and negotiating permission to call them consumed more engineering time than writing a single line of agent code.

How It Works

IBM’s Technology Lifecycle Services team built a multi-agent platform using LangGraph for orchestration and the Model Context Protocol for tool exposure. A Supervisor agent receives user queries and routes them to six independently deployed domain agents — Case, Asset, Documentation, Contract, and others — each owned by a separate team and running in its own environment.

Two protocols handle all inter-agent communication. Agent-to-Agent (A2A) enables remote agent calls with self-describing capability declarations, so agents can discover what their peers can do without tight coupling. MCP exposes enterprise tools as versioned service boundaries rather than in-process function calls, creating clear ownership at the edge of each team’s system.

The team chose LangGraph over low-code orchestration platforms after one architect drew a hard line: “low-code platforms are fast until they’re not. There is always a point where the business requirement doesn’t fit the tool.” Code-first was non-negotiable at enterprise scale.

Identity propagation runs end to end. User authorization tokens flow through every system hop — never substituted with service accounts. At organizational boundaries, tokens are exchanged rather than dropped, maintaining enterprise access controls across IBM’s federated identity systems. A shared AI CoE Agent Utils library provides base classes, tracing, authentication, and state persistence across all teams, eliminating the copy-paste drift that corrodes distributed codebases.

What This Means for You

Three production failures IBM documented publicly are worth internalizing before your own deployment:

Observability was treated as optional until it wasn’t. The team delayed distributed tracing, waiting for an internal platform that never materialized. They eventually adopted Langfuse, an open-source solution. The operational lesson: instrument from sprint one. In a multi-agent system where a Supervisor routes across six independently deployed services, invisible traces become production blind spots.

No baseline metrics made ROI indefensible. IBM’s TLS team reflected directly: “Without a baseline you can’t prove value to the people funding the work.” Agents that genuinely automate repetitive work become invisible — the work simply stops arriving. Invisible work can’t survive a budget review. Define measurable KPIs before building, not after launch.

Data access was the actual bottleneck. Identifying which internal systems held the relevant data, negotiating API permissions across organizational boundaries, and building secure wrapper services consumed more effort than all agent development combined. The insight is counterintuitive but consistent across enterprise deployments: the agent code is rarely the hard part.

By the time the team reached production on IBM Cloud Kubernetes Service — multi-region, multi-availability-zone — a predecessor system (Agent Assist) was already handling 91% of hardware error notifications automatically. The new platform built on that foundation, not around it.

Source: Tony Erwin

Content created with AI assistance and reviewed for accuracy.