Amazon's Kirorank: What Happens When You Gamify AI Adoption
Agentic AIAmazon’s internal “Kirorank” leaderboard — designed to accelerate AI adoption across its Kiro developer platform — lasted just long enough for employees to discover it could be gamed by directing AI agents at meaningless tasks.
The Metric That Ate Itself
When companies set adoption targets for new tools, they create the conditions for perverse incentives. Amazon’s Kiro platform ranked developers through an activity dashboard, and engineers did exactly what incentive theory predicts: they optimized for the score, not the outcome. AI agents were dispatched at low-value tasks solely to climb rankings — generating real cloud compute costs with zero business value.
Dave Treadwell, Amazon’s Senior VP for developer experience, acknowledged the damage directly: “Please don’t use AI just for the sake of using AI.” The dashboard, he noted, was built with good intentions. The result was measurable waste of expensive AI compute.
What Kirorank Actually Measured
Amazon has since replaced Kirorank with tracking “normalized deployments” — a metric that counts only AI-generated code that ships tangible value, rather than raw activity or token consumption. The distinction is critical. Raw usage metrics create performance theater where the appearance of AI adoption crowds out the reality of AI outcomes.
The pattern is not unique to Amazon. Meta ran a similar experiment tracking token consumption among engineers, with similar results: scores climbed while output quality didn’t. Both companies discovered that measuring inputs — how much AI is used — instead of outputs — what AI actually produces — is a recipe for gaming at scale.
The stakes compound fast. Amazon is targeting over 80% weekly AI adoption among developers, with $200 billion committed to AI infrastructure in 2026. At that scale, the gap between AI used and AI-delivered value is measured in billions of dollars of misallocated compute. Research into AI’s measurable impact on developer DORA metrics shows that teams coupling AI adoption to deployment frequency and change failure rates see materially better ROI — because the coupling forces attribution.
The So What
For any engineering organization building an AI adoption program, Kirorank is a precise map of what not to build. Raw usage, token consumption, and adoption-rate dashboards all create the same pressure: optimize the score, not the work. The defensible unit of measurement is deployed value — code that ships, decisions that close, workflows that wouldn’t have been possible without AI. A leaderboard doesn’t fix misalignment between AI tooling and actual work. Measuring outcomes does.
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →