GitHub Star History API: From 2,500 Requests to 24

Building the full star history for react/react, with about 251,000 stars, used to take about 2,511 API requests. GitHub’s new endpoint does it in about 24, and returns no stargazer identities at all. GitHub’s engineering write-up explains why the obvious fix, returning counts instead of people, didn’t make the query any cheaper. For platform teams designing internal APIs, it is a clean case study in where database work actually happens.

How It Works

Maintainers use star history to see whether a launch or release changed a project’s trajectory. Before this change, tools rebuilt the curve by paging through the stargazers endpoint, 100 people per page, keeping timestamps and discarding identities. That created two problems. Bad actors scraped public stargazer lists for spam and phishing, so GitHub restricted stargazer and watcher lists to repository administrators and collaborators, which broke tools like Star History. And the endpoint was expensive: a pagination bug made every request count all of a repository’s stars to work out the page total. In the week before the restriction, nearly 37,000 of those count queries ran slowly enough to be logged, averaging nearly 3 seconds, and database safeguards killed more than 1,000.

GitHub considered two designs before the one it shipped:

• Events without identities. Return starred_at without the user. It closed the privacy hole, but tools would still page through thousands of records.
• Aggregate, then paginate. Group stars into daily or weekly buckets and apply LIMIT and OFFSET. The response shrinks, but the database still finds, groups and sorts every star before LIMIT applies. For react/react, reading the full history that way would examine roughly 6 million rows across 24 requests.

The shipped design flips the order. Each request translates page and per_page into an indexed created_at range, filters to those weeks, then groups the matching stars into daily counts. A response covers up to 30 weeks. Across react/react’s full history, the database examines each star about once, roughly 251,000 rows. The new endpoint follows the same shape as the weekly commit activity API.

What This Means for You

The lesson GitHub draws first is the one most teams get wrong: a smaller response does not always mean a cheaper query. Pagination limits what leaves the database, not what the database does. The fix was to narrow rows before aggregation.

GitHub is also explicit about what remains. A week with an unusually large number of stars can still produce an expensive query, and calendar boundaries and removed stars affect what the endpoint returns, since the star model records only current stars, not a historical event log. The side benefit was real, though: comparing old and new paths surfaced the same count-on-demand pattern on the GitHub.com stargazers page, and switching it to a cached count took its slow queries, previously hundreds a week, to zero. Stack Archive covered another GitHub case of a narrower design beating a bigger one in GitHub Secret Scanning: Why a Small Model Beat the LLM Pipeline.

Source: The GitHub Blog

Content created with AI assistance and reviewed for accuracy.