Data Engineering Playbook: How GitHub Shrank Write Coordination to Move Its Last 0.00001% of Repositories Into Azure

A replicated write path that forces every transaction to line up behind its slowest healthy node will eventually cap the whole platform, and the durable fix is usually a change in coordination scope rather than a hardware purchase. That is the transferable engineering result coming out of a storage migration measured in months, not quarters. Over the past year, Git traffic expanded sharply and 99.99999% of Git repositories came to live in two places at once: GitHub data centers and Azure. The instructive part is the residue — 0.00001% of repositories, the busiest on the platform, all of them stopped by one physical constraint: a 25ms round trip.

Below, the migration is decomposed the way a data platform team would evaluate it: the sequencing decision, the two architectural changes, the measured throughput effect, the control-plane work that has to exist alongside them, and the failure modes to check before you schedule a cutover.

Sequencing the move: data first, architecture after arrival

The shape and the timing of the migration were dictated by the agentic shift: fast traffic growth, a hard ceiling on power capacity in the data centers and the nearby region, and physical supply chain constraints. At the start, remaining capacity plus every available expansion left enough room to migrate safely — and not nearly enough room to move slowly.

New Git scaling designs on cloud infrastructure had already been prototyped, so a combined “new architecture on the new platform” cutover was technically conceivable. It was rejected on risk grounds: swapping storage architecture and hosting platform together during rapid growth carried too much risk. The work was ordered instead — move data first, then evolve the architecture after arrival.

That ordering is a dependency argument rather than a stylistic preference. Git operations are a Tier 0 service: they are core GitHub functions, and nearly everything depends on repository data. Data, services, and dependencies therefore had to go first, because nothing that sits on top of a moving state store can be safely re-architected.

The move was framed by two principles in an earlier post: minimize coordination, and decouple storage from compute. The bridge between the old design and the new one is a single change in how Spokes serializes writes. That change pulls the busiest repositories into Azure, and it also opens a path to replicas without persistent state.

Pre-migration checklist for a state-store relocation

• Classify the workload tier. If the service is Tier 0 and everything downstream reads its state, the data movement is the critical path, not the redesign.
• Test your capacity margin against the fastest safe pace. Remaining capacity plus every available expansion has to cover the whole migration window; if headroom only supports a slow rollout, the plan is under-scoped.
• Confirm that traffic can follow the data. Most repositories keep five replicas to balance durability, availability, and scale; routing has to move request share as each replica arrives, rather than as an afterthought.
• Measure the write ceiling as a function of replica placement. A repository hosted in more than one region behaves differently from one whose replicas are all co-located.
• Document what the quorum actually waits for. “Majority to commit” and “waits for every healthy replica to answer, up to a timeout” are two very different latency profiles.
• Find the lock. Ask whether a write holds a scope-wide lock until commit; if it does, the slowest healthy replica is your throughput ceiling.
• Check whether cache memory and compaction compete for the same pool. If they do, read scaling is blocked until roles are separated.
• Plan the control plane. Placement, repair, and rebalancing between storage nodes need background systems before the topology starts changing under load.

The rollout, step by step

1. Land the first copy. A first repository copy landed in Azure Central US in August 2025, proving the path before anything was depended upon.
2. Move replica share, then let traffic follow. Today, most repositories have two of their five replicas in Azure, and Azure handles about half of Git traffic. For much of that period, Azure absorbed growth while the older data centers stayed level.
3. Route preferentially. Spokes uses preferential routing to reach a replica in the same region and availability zone as the request, lowering read latency. As each replica moves into Azure, the share of requests Azure can serve grows, until the final replica arrives and the old region stops receiving requests — every repository passes through this sequence.
4. Shrink the serialization scope. Move quorum coordination from the entire repository to individual references, which lifts the write ceiling enough to admit the remaining repositories.
5. Admit object storage to the quorum. Make Azure Blob Storage a quorum member through a virtual write replica so durability no longer requires a full persistent compute node per copy.
6. Scale the read fleet independently. Add virtual read replicas as a repository needs them, warming each from object storage instead of provisioning permanent nodes.
7. Close the remaining read classes. Move reconciliation in-line — fetching only the reference state and data that changed since a replica last caught up — so ephemeral replicas can serve every kind of read.

Root-cause analysis: what the 25ms tax actually costs

Under the current architecture, a repository hosted in more than one region has a lower ceiling for write throughput. The Spokes quorum only needs a majority to commit a write, yet it waits for every healthy replica to answer, up to a timeout, before continuing. Slow replicas therefore delay every write instead of merely falling behind.

The compounding factor is lock scope. Because each write holds a repository-wide lock until commit, the slowest healthy replica also fixes the throughput ceiling. Once a replica sits in the target Azure region, it is about 1,000 miles from the older data centers, so the cross-region delay is a 25ms round trip, and a write needs multiple trips. A repository already near its ceiling cannot absorb that penalty on every serialized write; it would fall further behind with every push.

Read this as a queueing problem, not a bandwidth problem: the latency does not just add time to one request, it serializes the whole write path behind the slowest participant.

Change 1 — serialize per reference instead of per repository

Minimizing coordination means not serializing what does not need to be serialized. Git consistency rules attach to each reference separately: after the server accepts a reference update, no client should later see an older value, only that value or a newer one. The rule applies whether the update is a fast-forward, a force push, or a delete. In most cases, different references can be updated independently.

So the quorum scope shifts from the entire repository to individual references.

Serialization scope	Queue behavior	Write overlap	Effect on the throughput ceiling
Repository-wide	one write in flight at a time	none	set by the slowest healthy replica, multiplied by cross-region round trips
Per-reference	only shared references wait	different references overlap	rises in proportion to the number of references updated concurrently

Holding a repository-wide lock through an update, including round trips, makes the quorum behave like one unordered request queue with one write in flight at a time; each queued write lasts as long as the slowest healthy replica. As the write rate rises, the queue lengthens and each write waits longer than the previous one.

Per-reference serialization narrows waiting to writes that contend on the same reference. In practice, most repository writes touch different references — agents and developers pushing their own branches — so they no longer block one another. A cross-region latency cost remains for each write, but different references can move at the same time, removing the single repository-wide bottleneck.

The measured effect matters for capacity planning: in internal testing with realistic traffic, writes to different references consistently saw 3–4x higher throughput at lower latency. Those adjustments lifted the write throughput ceiling in proportion to concurrent references and moved the last remaining repositories into Azure.

Change 2 — split the read path from the write quorum

Workload skew drives this one. Most repositories are heavily read-skewed, with somewhere between 100 and 1,000 reads for every write. Today, reads and writes use the same replicas, so they compete for the same resources and cannot be scaled independently. Reads could benefit from far more in-memory cache, but that memory is also needed for compaction. Once the roles are split, each can have its own configuration.

Decoupling storage from compute is what makes the split legal: once repository data is durably stored in Azure Blob Storage and updated with each incoming write, read traffic can be separated from the write quorum, and the compute for each can be scaled independently.

The initial move makes object storage a quorum member through a virtual replica. The compute standing for it is the first fully ephemeral node: it bridges to object storage, streams incoming writes, and serves no reads. Because it is part of the quorum, a write is not acknowledged until the data is durable in object storage.

That adds a little write latency. As with cross-region round trips, per-reference serialization prevents the latency from becoming a throughput bottleneck: the virtual write replica coalesces transactions on disjoint references into one set of network I/Os to object storage, so a busy repository pays for one round trip across many concurrent writes instead of one per write. For now, the virtual write replica also performs compaction, and read replicas inherit the compacted state.

With that replica in place, many virtual read replicas can be scaled out as a repository needs them. Each read replica warms its cache from Azure Blob Storage and starts taking read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state against it in the background. In internal testing, the distributed load testing harness routinely found ceilings in the current architecture; against virtual read replicas, the harness hit its own limits first.

What is still bounded after both changes

Two caveats belong in any rollout plan built on this pattern. Background reconciliation suits some read workloads but not all; serving every kind of read requires moving that reconciliation in-line. And although per-reference serialization has increased write throughput incrementally, the quorum remains the bounding factor for further growth.

The control plane has to be budgeted alongside the data path

Today GitHub rests on the Spokes quorum. Over the years, Spokes has become much more than a coordinator: an intelligent routing and load balancing layer; a unified API used by internal clients to access and transact repository data at well over 100 billion operations per day; and background systems called “sweepers” that handle repository placement, repair, and rebalancing between storage nodes. Most of the work under the Spokes umbrella also fits the new architecture.

The exception is the commit protocol itself, and it is the part platform owners should plan around rather than assume away: the commit protocol that carried GitHub to a billion repositories will not carry it to the next billion and beyond.

Common misconceptions

• “A majority commit means lagging replicas are harmless.” Not while the quorum waits for every healthy replica to answer, up to a timeout: slow nodes delay every write.
• “Cross-region latency only hurts tail p99.” Under a repository-wide lock it also sets the throughput ceiling, because a write needs multiple trips.
• “We can switch hosting platform and storage architecture in one window.” Doing both during rapid growth carries too much risk; order them — data first, then architecture.
• “Separating reads from writes is free.” Making object storage a quorum member adds a little write latency before it removes anything.
• “Ephemeral replicas serve every read immediately.” They start by reconciling in the background, which suits some read workloads but not all.
• “One more scaling design is enough.” The next limit is write throughput: every write still waits on replicas that each hold a full copy of the repository, while very little of a write actually needs ordering. Git objects are immutable and named by their contents, so they can be stored without anyone agreeing on them; what remains is small, and the hard part is ordering it as quickly as possible without moving the bottleneck elsewhere.

FAQ for data platform teams

Why did the last 0.00001% take the longest?

Because they are the busiest repositories on the platform, and they were already near the write-throughput ceiling. A ceiling that is already saturated cannot absorb an added cross-region penalty on every serialized write — each push would leave the repository further behind.

Does per-reference serialization remove the cross-region cost?

No. A cross-region latency cost remains for each write. What disappears is the repository-wide bottleneck: writes that touch different references proceed at the same time, so the ceiling lifts in proportion to the number of references updated concurrently.

Which operational problems show up as second-order effects?

Placement, repair, and rebalancing between storage nodes have to run in the background, and the redesign forced a rethink of where durable reference state lives and how compaction and garbage collection scale.

Can each step ship value before the end state exists?

Yes — that was the operational argument for the ordering. Agent traffic is not slowing, and each step gave repositories more headroom as it landed, without waiting for the end state.

Takeaways, and what to watch next

Three things carry over to any pipeline or state-store relocation: order the work by dependency rather than by architectural elegance; treat lock and quorum scope as a first-class scaling knob, measured against your slowest healthy node; and store durable state once, then scale read compute independently instead of buying permanent replicas. The remaining challenge is ordering the small mutable part of a write as fast as possible without relocating the bottleneck — designing that on a whiteboard is one thing, doing it underneath a billion repositories that developers and agents write to around the clock is another.

That end state, what it does for write throughput, the second-order problems that had to be solved to reach it, and why a small consensus group remains somewhere in the stack, is due next week — at GitHub Universe and in the final post in this series. Brian Celenza contributed to this post.

By David Nakamura for AI & Data News (AIdataNewsGlobal), DataEngineering desk. Editorial contact: editorial@aidatanews.com.