Four Engineering Pillars for Reliable Agentic AI in Production
Agentic AIThe enterprise AI conversation centers on model quality and prompt engineering. Bayer’s PRINCE system — a production agentic platform live since early 2024 — offers a corrective: “Production-ready agentic AI is not only about better models or better prompts.”
The Framework
PRINCE (Pharmaceutical Research Intelligence and Comprehensive Navigation Engine) gives Bayer researchers access to decades of preclinical safety PDFs, structured study metadata, and regulatory knowledge across data silos. Writing on Martin Fowler’s site, the engineering team distills the production experience into a framework built on four pillars: context engineering, harness engineering, layered reflection, and structured hybrid retrieval.
The unifying principle: “reliability comes from engineering both the context the model sees and the harness within which the model acts.”
Context Engineering
The instinct when agents underperform is to give them more context. PRINCE found the opposite: “larger context windows did not remove the need to be selective about what each agent sees.”
The solution is stage-specific context allocation. The planning stage receives a different context package than the research stage, which differs from the reflection stage, which differs from the writing stage. Each specialized agent sees only what it needs for its function. The result is less context pollution, more evaluable outputs, and significantly easier debugging — because when something fails, the failure is traceable to a specific stage and its specific inputs.
Harness Engineering
This is the infrastructure layer that production agentic systems consistently underinvest in. PRINCE uses LangGraph for orchestration with workflow state persisted in PostgreSQL via LangGraph’s checkpointer. Application-level state lives in DynamoDB.
The harness provides automated retries at both the LLM call level and the logical workflow node level, LLM fallbacks across OpenAI, Anthropic, Google, and open-source providers, and error context surfaced to agents so they can adjust trajectory rather than simply failing. Node-level recovery means a failed workflow segment restarts without replaying the entire pipeline. SQL query generation gets up to three attempts, with each failure’s error message fed back to inform the next.
The team built this because provider unavailability and malformed LLM outputs are regular occurrences at production scale — not edge cases.
Layered Reflection
The most differentiated element of PRINCE is a three-loop reflection system where each loop addresses a different failure mode.
Process reflection (Think & Plan) evaluates workflow trajectory before execution. Inspired by Anthropic’s Think tool, this stage determines which tools to invoke from a growing toolset and sequences multi-step actions before they run — catching planning errors before they propagate.
Data reflection (Reflection Agent) compares retrieved context against the original query to identify evidence gaps. When retrieval is incomplete, the agent generates follow-up questions triggering additional retrieval passes, iterating until data sufficiency is confirmed before synthesis begins.
Draft reflection (Writer Agent) runs an internal review loop after a response is drafted. The agent validates completeness against the original question and issues targeted revision instructions when the draft falls short — enforcing formatting and citation requirements before the response exits the workflow.
Three reflection checkpoints, each catching failures invisible to the others.
Structured Hybrid Retrieval
Enterprise data doesn’t live in one form, so the retrieval layer combines RAG over unstructured PDFs with Text-to-SQL over structured study metadata. The RAG component involves six steps: keyword extraction, metadata filter generation via few-shot prompting, query expansion (five semantically similar variants per query), weighted hybrid search (0.7 semantic + 0.3 keyword), reranking via bge-reranker-large to select the top 7 from roughly 20 candidates, and chunk-level citation generation.
Metadata filtering is the scalability mechanism: filtering by study type, compound, date range, and similar attributes pre-reduces the candidate space from millions of documents to hundreds before vector similarity runs. The vector store is Amazon OpenSearch Service.
What the Team Got Wrong
The SQL review step was removed because the reviewing LLM “sometimes incorrectly flagged valid queries as erroneous” — a reminder that reflection loops can themselves introduce failures. The monolithic Researcher Agent became difficult to manage as domains expanded, prompting a shift to domain-specific sub-agents (toxicology, pharmacology) that remains in progress. All outputs from regulatory document drafting workflows “are intended for expert review; final submissions are authored and approved by qualified personnel.” The system augments expert judgment; it doesn’t replace it.
The So What
The four-pillar framework gives teams a concrete diagnostic: when an agentic system fails in production, the root cause traces to context pollution, harness gaps, insufficient reflection, or retrieval quality — and each has a structural fix. The layered reflection pattern in particular is underimplemented in most production systems relative to how often output-quality failures trace to insufficient self-review within the pipeline. For teams building multi-step agents for knowledge-intensive domains, PRINCE is one of the more rigorously documented production references available.
Source: Martin Fowler
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →