Scalable Frequency- and Length-Aware Subdocument Deduplication for Large Language Model Pretraining

2026-08-04

AI Summary

This work addresses the pervasive issue of sub-document-level redundancy in large-scale pretraining corpora, which existing methods struggle to identify efficiently across distributed shards while flexibly preserving redundant copies. The authors propose a scalable sub-document deduplication framework that decouples duplicate detection from copy retention: it leverages natural boundary segmentation, normalized exact hashing, and distributed aggregation to identify duplicate groups, and introduces—for the first time—a frequency- and length-aware adaptive copy retention strategy that overcomes the limitations of fixed heuristic rules. Experiments on FineWeb-Edu and web corpora containing code demonstrate that the proposed approach significantly enhances model training performance, underscoring the importance of explicit control over copy retention during deduplication.

Abstract

Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging. At corpus scale, suffix-array-based methods are commonly applied independently within shards, leaving cross-shard duplicates undetected and making the resulting retention behavior sensitive to the sharding configuration. Hash-based methods enable global exact duplicate counting, but often rely on fixed copy-retention policies that cannot accommodate heterogeneous repetition patterns. We propose a scalable subdocument deduplication framework that decouples duplicate detection from copy retention. It identifies duplicate groups through natural-boundary segmentation, normalized exact hashing, and distributed aggregation, and then applies an explicit frequency- and length-aware retention policy that allocates an adaptive copy budget to each group, retaining more copies of low-frequency or short repetitions while more aggressively deleting high-frequency or long ones. Experiments on FineWeb-Edu and a code-containing web corpus show that models trained on data processed by our method achieve the best overall performance among the evaluated settings. These results underscore the importance of explicit copy-retention control.

Problem

• subdocument deduplication
• large language model pretraining
• duplicate retention
• frequency-aware
• length-aware

Innovation

• subdocument deduplication
• frequency-aware retention
• length-aware retention
• scalable deduplication
• duplicate detection