Was It Never Collected, or Rewritten Away? A Commit-Provenance Dataset Separating Ingestion Gaps from Upstream History Edits across the World of Code

2026-07-02

AI Summary

This study addresses the frequent misattribution of missing commits in open-source code mirrors, which are often ambiguously ascribed to either upstream force-push deletions or incomplete mirror ingestion. For the first time at full scale, this work disentangles these two root causes by integrating GHArchive event streams with the World of Code database, leveraging reference-chain backtracking, cross-source commit graph comparison, and a structured force-push detection algorithm. The resulting provenance dataset spans 1.1 billion commits, each labeled as “ingested,” “rewritten,” or “unobserved.” The analysis reveals that 53.35% of commits are ingested, 6.47% are rewritten, and 40.18% remain unobserved. The study releases a force-push dataset comprising 167 million events and statistics from 78 million repositories, correcting an underestimation of developer productivity by 10.82% and substantially improving the accuracy of mirror completeness assessment and contribution accounting.

Abstract

Any global mirror of open-source version control is incomplete, but the reasons a commit is missing are not interchangeable: a project may have force-pushed it away (so it no longer exists upstream), or the mirror may never have ingested it (a true collection gap). We release a commit-provenance dataset that separates these cases at scale by comparing two views of the same commit graph: the GHArchive event stream, a historical witness of what GitHub advertised at push time, and the World of Code (WoC) V2604 object database, an accumulated union of periodic fetches that never deletes a collected commit. Walking each reference's PushEvent chain reconstructs force-pushes structurally (a before that is not the prior head breaks the fast-forward chain; no recorded flag exists across eras), and joining every advertised commit against WoC membership yields a three-way label. Over 1,118,116,350 advertised commits, 53.35% are present in WoC, 6.47% are rewritten (orphaned by a later force-push, an upstream edit and a correct absence), and 40.18% are never-ingested (the candidate collection gap). About one missing-commit case in fifteen is a rewrite the project erased, not a mirror gap. We release the force-push witness (166,710,831 events over 19,926,250 repositories) and a per-project rollup (78,125,788 repositories; force-pushing observed in 25.47%, a rewritten commit in 12.85%). We treat the 40% never-ingested share as an upper bound on the collection gap and state five reasons it overcounts. The dataset makes mirror-completeness reporting honest, flags rewritten commits as duplicate patch sources for contribution counting, and corrects a 10.82% corpus-wide undercount that history editing imposes on commit-based productivity, with a released per-project correction factor. All artifacts ship as a self-contained, independently hosted replication package keyed to WoC V2604.

Problem

• commit provenance
• force-push
• collection gap
• version control
• history editing

Innovation

• commit provenance
• force-push detection
• mirror completeness
• history rewriting
• dataset curation