AWS Extends Its DevOps Agent to Diagnose EKS Node Issues via Custom MCP
Cloud Platform EngineeringAI agents hit a consistent wall in production incident response: they can reason about what they can see, and OS-level node data on EKS workers was not in scope. That gap just closed.
What AWS Built
AWS’s DevOps Agent already handles a broad range of incident investigation tasks within its native integrations, but EKS node-level diagnostics — iptables rules, CNI configuration, kernel logs, kubelet state — exist below the API surface where the agent normally operates. Giving the agent shell access to worker nodes introduces security and auditability problems that are worse than the diagnostic gap.
The solution is a custom MCP server that mediates between the DevOps Agent and the node OS. The agent sends investigation requests to the MCP server; the server dispatches SSM Automation runbooks to nodes running SSM Agent; those runbooks collect over 20 log sources per node; the results are uploaded to encrypted S3, pre-indexed with severity classifications (Critical, Medium, Normal), and exposed back to the agent as queryable MCP tools.
Authentication runs through Amazon Bedrock AgentCore Gateway using OAuth 2.0 via Cognito. The agent never touches the nodes directly — every action is IAM-controlled and auditable through SSM Automation’s execution history.
What the Architecture Enables
The design separates collection from reasoning. SSM handles the privileged collection work with its existing IAM controls; the MCP server handles the indexing and tool surface; the DevOps Agent handles the reasoning. Each layer does only what it is suited for.
In the walkthrough scenario documented by AWS, the agent investigated DNS resolution failures on an EKS cluster. It collected logs, then ran four parallel deep-dive tasks: iptables and route table analysis (8 MCP tool calls), network error search (15 calls), healthy node comparison (13 calls), and OOM/pod status check (19 calls). Total autonomous investigation time: approximately 14 minutes. Root cause identified: two CRITICAL iptables FORWARD chain DROP rules and a blackhole route for the service CIDR were blocking pod-to-service communication.
The healthy node comparison capability is particularly useful — it gives the agent a differential baseline rather than asking it to reason about whether a given log state is abnormal in isolation.
The So What
The pattern here is more significant than the specific EKS use case. Custom MCP servers as the privileged collection layer — with structured, severity-classified outputs exposed as agent tools — is a reusable architecture for any scenario where an AI agent needs access to data that exists below its normal API surface. OS logs, hardware telemetry, audit trails, network captures: any of these can be made agent-accessible through the same pattern without granting the agent direct privileged access. AWS has demonstrated the reference implementation; the design applies well beyond EKS.
Source: AWS Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →