Kong Gateway: How Agentic AI Fixed 12 Flaky Tests in 1.5 Weeks
Agentic AIKong Gateway’s test suite runs 34,000 test cases across two architectures—a 23.5-hour execution on a single machine. Flaky tests, which fail intermittently without code changes, create compounding delays: engineers wait 30 minutes to an hour for CI queues before tests even start. Fixing them manually requires deep codebase familiarity and tedious repeated reruns that humans reliably deprioritize.
What Happened
Kong’s engineering team deployed a three-tier agentic AI system using Claude Code to target the 15 flakiest tests in the suite. Over 1.5 weeks, the system fixed 12 of 15 tests, spending approximately 4–5 hours per test (mostly in verification). Two genuine production bugs emerged during investigation that would not have been found by traditional debugging approaches.
The architecture used three specialized agents:
- fix-flakes (Claude Opus): Orchestrator that downloads failed test logs from Datadog CI exports
- flake-fixer (Claude Opus): Investigates failures across log files and source code, produces patches—typically completing in under 10 minutes
- flake-verifier (Claude Haiku): Validates proposed fixes by analyzing unstructured CI logs across multiple reruns, at lower cost
The verification loop is self-correcting: if a fix fails verification, flake-fixer spawns again with combined context from the previous attempt. Token usage ran 100–200k per test, with Haiku absorbing the high-volume verification tasks to control cost.
Why This Architecture Works
Two design decisions matter more than the model selection. First, agents write single-purpose throwaway scripts rather than relying on generic pre-built tools. When an agent needs to parse a specific log format or correlate test failures across multiple CI runs, it writes the parser inline for that task. This eliminates the abstraction overhead of building a generic harness that may or may not fit each new failure pattern.
Second, context passes between cycles as summaries, not transcripts. Only the essential findings from each investigation cycle carry forward, preventing context bloat from degrading subsequent reasoning. This is an explicit counter to the common failure mode of agentic systems where accumulated context degrades mid-task coherence.
The unexpected production bug discovery is the most interesting result. In one case, a five-year-old sorting nondeterminism in conf_loader left lua_shared_dict directives in unpredictable order—a bug invisible to traditional testing but exposed when an agent traced a flaky test to its actual root cause.
The So What
Flaky tests are a near-universal problem in large test suites, and they’re an ideal first target for agentic engineering precisely because the blast radius is contained—you cannot make production code worse by fixing a test. Kong’s case suggests the pattern scales: a three-tier architecture with a high-capability reasoning model, a lighter verification model, and minimal context carryover between cycles handles this class of problem efficiently.
The relevant number for teams evaluating this approach: 100–200k tokens per test. At current Claude pricing, that’s a one-time cost to eliminate a recurring CI drag. For teams where a flaky test wastes 30+ minutes of engineer queue time per occurrence, the math closes quickly.
Source: Kong Engineering Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →