Agentic AI Overwhelmed CI — Test Selection Cut P95 Queue Time by 92%
Agentic AIA solo developer’s hobby project grew to 500,000 lines of code — not because they hired a team, but because coding agents kept writing. Then the CI system broke under the weight of it all.
What Happened
When you combine high-velocity coding agents with a single CI runner, you get a queue. In a detailed post-mortem, the developer describes a loop where agents submitted changes every few minutes while each CI run took roughly 20 minutes. By the time one run finished, several more commits were waiting. The P95 queue time swelled to 7 hours and 35 minutes.
The tempting fix — add more runners — would treat the symptom. The developer instead implemented intelligent test selection: a system that identifies which tests are actually affected by each change and runs only those.
The implementation combined two data sources. TypeScript’s compiler provided a dependency graph showing what code was structurally affected by a change. Runtime traces from previous test runs showed which parts of the application each test actually exercised. Together they allowed the system to flag affected tests with high confidence — and explicitly flag uncertainty rather than assume safety when the evidence was ambiguous.
Why It Matters
The P95 queue time fell from 7 hours 35 minutes to 35 minutes. Wall clock test time dropped from roughly 21 minutes to 13 minutes (a 40% improvement), and the end-to-end lane from 12 minutes to 4 minutes (65%). But the numbers reveal something more interesting: most of the gain came from eliminating queue time, not from running faster tests.
This reframes how teams should think about CI at agent velocity. A slower test suite with smart selection outperforms a fast suite running everything. The bottleneck isn’t compute — it’s the ratio of commit frequency to verification time, and agents skew that ratio toward verification debt faster than human developers ever did.
The broader pattern here is consistent: agentic workflows don’t just accelerate development, they expose infrastructure assumptions that were fine at human commit velocity. CI queuing is one. PR review throughput is another.
What This Means for Builders
Teams adopting coding agents should treat CI queue depth as a leading indicator, not a trailing complaint. The traditional response — scale horizontal runners — works up to a point, but it doesn’t change the fundamental ratio. Test selection does.
The architecture described here is reproducible: collect runtime coverage data (most modern test runners support this), build a dependency graph from your compiler or language server, and combine them to scope each run. Conservatively flag ambiguous cases as needing full tests rather than optimistic skips — the goal is reducing unnecessary runs, not gambling on coverage gaps.
The So What
At agent velocity, your CI pipeline becomes the constraint faster than you expect. The answer isn’t more runners — it’s more selective verification. Building or adopting test selection before agents saturate your queue is cheaper than retrofitting it afterward.
Source: Human Systems
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →