AIdataNewsGlobal DataEngineering Guide: GitHub Star History from 2,500 requests to 24 without exposing stargazer identities
A platform team rarely learns more from an API rewrite than from its cost curve. In the star-history case, the same demand for week-level growth signals can be served in two very different ways: a list-style path that behaves like a full scan, or a windowed aggregate that collapses the workflow from 2,500 requests to 24 and returns no stargazer identities.
Engineering context: why the original star data was not a safe metric surface
Maintainers wanted an approximate read on interest: did a launch, release, or event change the slope? That job did not require usernames, but older tooling derived it from GitHub API records because each starred_at timestamp came attached to a person. The pattern created two operational problems. Bulk stargazer access became an abuse path on GitHub.com for spam or phishing, and highly starred repositories made the database touch large star sets just to serve one page.
The permission fix came first: repository administrators and collaborators retained list access, while unauthorized callers could receive an empty result or a 403 Forbidden. That broke clients that did not even display the identities they were reading. Conversations with Star History and maintainer Tianzhou Chen showed the required contract was counts over time, not user data.
Tianzhou Chen, star-history.com maintainer and Bytebase co-founder, summarized the collaboration: when the change disrupted Star History, users rallied around the project; GitHub’s Community team listened and connected the project with product engineering, who examined the use case and co-designed a purpose-built endpoint.
Baseline cost model: how a list endpoint became a history workload
The original query answered who starred a repository. Pagination returned at most 100 people, and a custom media type added the creation timestamp. Because the star table stored the current set rather than an append-only event log, un-starring erased the row; therefore building a full curve meant processing the entire current star population. For react/react, a repository with about 251,000 stars, this meant about 2,511 requests before rate limits pushed many tools to sample.
Page serving carried a hidden second cost. A pagination bug caused each page request to count all current stars to derive the page count. In the week before we restricted access, logs showed nearly 37,000 slow count queries, averaging nearly 3 seconds; database safeguards killed more than 1,000, adding request timeouts. After we restricted access, the database served about 60% fewer stargazer requests, and the total time spent on the count query fell by about 80%.
Architecture options considered for a low-cost historical count
| Option | Privacy outcome | Query consequence | Engineering verdict |
|---|---|---|---|
| Keep event rows, hide identities | Stops username leakage | Consumers still paginate many records; the endpoint remains shaped as a list of people | Fast mitigation, poor long-term fit |
| Aggregate first, paginate later | Returns counts instead of identities | Database finds all stars, groups buckets, sorts buckets, and only then applies OFFSET/LIMIT; for react/react the full read could touch roughly 6 million rows because each of the about 24 requests reprocessed all stars | Shrinks response size but not compute cost |
| Filter by calendar week before grouping | Returns daily star counts for a bounded window, up to 30 weeks | An indexed created_at range narrows rows first, then daily buckets are grouped; full history can be served with about 24 requests, and the database examines each star about once, or roughly 251,000 rows | Selected pattern for weekly trend inspection |
Implementation steps: building the time-filtered aggregation path
- Separate the data product from the raw record: define that the consumer needs dates and counts, not social identities.
- Remove the high-risk surface first: restrict identity-bearing lists to authorized roles, and return empty or 403 Forbidden for unauthorized callers.
- Choose the query order before choosing pagination mechanics: apply a time-range predicate first, then aggregate, then shape the response.
- Map page and per_page parameters to indexed calendar-week ranges. Each request can cover up to 30 weeks of daily star counts, and increasing page walks backward through repository history.
- Align the response shape with existing time-series APIs, such as weekly commit activity, so client tooling can compare repository growth across signals.
- Monitor residual hot paths: a week with unusually many stars can still make an otherwise windowed query expensive.
Operational tradeoffs after the redesign
The shipped pattern improves predictability because the main path no longer queries every current star. It still does not erase the original data model: the repository stores current stars, not a complete event history, so removed stars, calendar boundaries, and exceptionally dense weeks remain part of the contract. If a single week contains an extreme spike, the time filter narrows the scan but does not bound the number of matching rows.
Comparing the old and new paths also exposed additional star-counting work. GitHub.com’s repository stargazers page was switched to the repository’s cached star count; slow queries from that page, which had numbered in the hundreds each week, dropped to zero. A similar fix was shipped to other stargazer API endpoints.
Common misconceptions for data engineering readers
- A smaller JSON body proves the query is cheaper. It does not. LIMIT and OFFSET can reduce returned rows after grouping and sorting are already complete.
- Historical trends require individual identities. The Star History use case required counts and dates; identities were an artifact of the old list endpoint.
- Aggregation is enough if pagination exists. If pagination is applied after full aggregation, popular repositories still pay a full scan cost.
- Privacy and performance must be traded. In this redesign, returning counts and filtering by time improved both for the primary workload.
FAQ for evaluating a similar rebuild
Q: Does the new endpoint completely remove expensive star counting?
A: It removes the default full-history scan from the main star-history path, but dense weeks can still be costly, and other endpoints required separate fixes.
Q: How many page calls did react/react require before and after?
A: The old identity-list reconstruction took about 2,511 requests; the week-window design took about 24 requests, and the overall result is 2,500 requests becoming 24.
Q: Where can implementation feedback go?
A: The GitHub Community discussion is a channel for sharing feedback.