How Datadog Scaled Git Serving 20x Without Slowing CI Down
Enterprise Developer PlatformsMost teams treat Git infrastructure as solved infrastructure — until CI starts failing at scale. Datadog’s GitRetriever case study describes what happens when a monorepo CI pipeline hits 20x traffic growth: the standard scaling approaches break down, and the solution requires rethinking how Git objects move through the build system entirely.
How It Works
The problem at Datadog was a thundering herd pattern. CI jobs trigger Git fetches against large monorepos; as the number of concurrent jobs grows, those fetches concentrate CPU load on a small set of Git servers. Traditional horizontal scaling made things worse — adding nodes increased replication overhead, creating more synchronization work precisely when the system was already stressed.
GitRetriever solves this with a two-tier architecture:
Mirrors — a small fleet that polls GitHub continuously, distributing pull work across branches rather than making monolithic repository fetches. Mirrors stay current without generating the expensive full-clone operations that were saturating the old system.
Relays — an autoscaled read tier that serves CI jobs directly, pulling packfiles from mirrors and reusing them without regeneration. The key insight is content-addressing: because Git objects are identified by their content hash, relays can trust data from mirrors without any coordination. There’s no multi-writer consensus, no distributed locking, no consistency protocol — just disposable nodes that rebuild from the mirror layer when needed.
The result across four months: median latency held at 40ms while serving over a billion Git requests and hundreds of terabytes of code. CPU load on the old backend dropped three to four times despite growing CI activity. Synchronization latency fell from several seconds to a few hundred milliseconds.
What This Means for You
The architecture principle that makes GitRetriever work generalizes beyond Git. Content-addressing turns a coordination problem into a caching problem — and caching is a problem the industry has solved at scale many times over. The same logic applies to any read-heavy, write-sparse infrastructure that suffers from thundering herds: artifact registries, package caches, build result stores.
The “API-first query layer” point from Datadog’s write-up is the other transferable lesson. Moving CI jobs from expensive full clones to lightweight API queries — even when those API queries are still Git protocol under the hood — proved unexpectedly impactful. Anywhere a team defaults to “fetch everything” when “fetch what’s needed” would serve, this pattern applies.
For platform teams running CI/CD pipelines with Datadog observability, GitRetriever also demonstrates how telemetry shapes architectural decisions: the thundering herd pattern was visible in trace data before it became a production emergency.
The So What
If Git latency is not a problem for your CI today, you may not need GitRetriever’s architecture. But if you’re managing a monorepo CI pipeline that grows predictably — developer headcount, commit frequency, or test coverage expansion — the two-tier mirror/relay pattern is worth understanding before you hit the wall. Datadog’s write-up is unusually candid about where each approach failed before they found what worked, which makes it useful as engineering documentation beyond the Datadog-specific context.
Source: Datadog Engineering Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →