Building Production Agentic AI: IBM's Hardest-Learned Lessons

Most agentic AI architecture posts spend 80% of their word count on agent frameworks and 20% on what actually slowed the team down. IBM’s account of its production TLS platform inverts that ratio — and the inversion is the lesson.

How It Works

The IBM team picked LangGraph over low-code orchestration tools deliberately. The reasoning: LangGraph “stays close to the mechanics rather than hiding them,” enabling explicit state management and node-level streaming control that evolving, complex workflows require. Low-code alternatives abstract away exactly the control surfaces production teams need.

The architectural spine: a Backend-for-Frontend layer using Server-Sent Events for streaming handled authentication and rate limiting at the perimeter. Six independently-built specialist agents coordinated through two protocols. The A2A (Agent-to-Agent) protocol enabled independent deployment across teams and environments — one specialist agent ran in a separate environment operated by a different group entirely. The MCP (Model Context Protocol) abstracted tool access behind service boundaries, letting tool owners evolve implementations without breaking downstream agents.

A shared Python library, AI CoE Agent Utils, became load-bearing infrastructure before the team fully realized it: base classes, distributed tracing, authentication middleware, and the core agent harness. What started as boilerplate reduction became the connective tissue across six independently developed components.

Identity propagation turned out to matter more than any agent design choice. The principle: “propagate or exchange, never substitute a service account.” Token exchanges with downstream systems like Salesforce preserved enterprise authorization models rather than centralizing access control into a service identity that bypassed the organizational trust model.

What This Means for You

The post names three failures that cost the IBM team time and money.

Observability delayed. The team eventually adopted Langfuse, an open-source observability tool, rather than waiting for internal platform roadmaps. By then, the absence of reliable tracing had already shaped early architectural decisions without data. The lesson is not “use Langfuse” — it’s that waiting for perfect internal observability tooling is more expensive than adopting something imperfect now.

Data access, not agent code, was the constraint. Identifying systems of record, negotiating API access, and building wrapper layers consumed more engineering effort than the agent code itself. This is the hardest thing to communicate to non-technical stakeholders who assume the AI model is the bottleneck.

Evaluation came after shipping. The absence of rigorous baselines and business KPIs before launch meant the team couldn’t prove value or identify which agents delivered actual returns. The evaluation framework should exist before the first agent goes to production, not as a retrospective exercise.

These aren’t IBM-specific failures. They’re the pattern across enterprise agentic deployments — which is what makes the post valuable as a reference beyond the specific technology choices made.

As Stack Archive has covered in depth, the gap between agentic AI capability and production-grade deployment is still significant — and data access is consistently where that gap shows up first.

Source: Tony Erwin

Content created with AI assistance and reviewed for accuracy.