🤖 AI Summary
This study addresses the underestimation of truncation loss in web-PDF corpus evaluation when relying solely on document-level metrics. For the first time, we quantify the statistical discrepancy between document counts and token distributions. Through large-scale parsing, Gini coefficient analysis, and cross-corpus validation, we demonstrate that merely 3% of documents contain half of all tokens, while truncation results in 55–62% text loss with existing recovery tools achieving less than 15% restoration. These findings confirm severe distortion in single-unit document statistics. Consequently, this work proposes a dual-unit reporting standard incorporating both document and token metrics, which significantly enhances the accuracy and scientific rigor of corpus evaluation, establishing a new benchmark for PDF data processing.
📝 Abstract
PDF corpora advertise their size in tokens but compute every rate they publish (coverage, OCR routing, re-fetch recovery, language mix) per document, and none decomposes its token total. The two units diverge sharply. On CC-MAIN-2021-31-PDF-UNTRUNCATED (7.9M web PDFs, 32.6B tokens), 3.02% of text-bearing documents hold half the tokens (Gini 0.807); documents over 50 pages are 5.00% of the corpus but 53.53% of its text. The PDFs produced by a TeX{} toolchain are 1.66% of documents and 4.05% of the text. The clearest casualty is Common Crawl's truncation cap: it affected 23.06% of documents and 63.08% of the text. Reconstructing the truncated files and extracting both versions, two widely used libraries recover 11.4% and 1.4% of that text; between 72% and 97% of affected documents yield nothing; roughly 55--62% of the corpus's text is lost. Under the 5 MiB cap adopted in March 2025, 30.19% of tokens would still be truncated, and recovery on those documents rises only from 3.3% to 13.2%. We recommend that corpus statistics be reported in both units: documents and tokens.