Auditing Chinese Web-scale Corpora via Sampled BPE Token Statistics

2026-08-11

AI Summary

This study addresses the challenge of detecting subtle and rapidly evolving contamination in Chinese web corpora, which traditional auditing methods struggle to identify efficiently at fine granularity. To tackle this issue, the authors propose Sampled-BPE, a lightweight auditing pipeline that trains a Byte Pair Encoding (BPE) tokenizer on small-scale samples to enable token-level identification and analysis of contaminated content for the first time in Chinese corpora. By integrating sampled training, token-level statistics, and hierarchical data organization, the method demonstrates strong efficacy across 11 open-source corpora and six Common Crawl snapshots. It achieves a 148.4× speedup and 35.8× lower memory usage compared to baseline approaches, with only a 4.25% relative error increase. The work also releases a traceable dataset comprising over 660,000 contaminated Chinese tokens.

Abstract

Chinese web pollution has surfaced in LLMs, motivating audits of upstream Chinese corpora. However, auditing such corpora faces three challenges: (1) their web-scale size makes full scan costly; (2) prior analyses are often too coarse to expose token-level pollution; (3) Chinese web pollution is implicit and rapidly changing. We propose Sampled-BPE, a lightweight token-level auditing pipeline that sample a small subset and train BPE tokenizer to surface polluted tokens. Experiments show that Sampled-BPE preserves usable estimates while substantially reducing runtime and memory: a 148.4 $\times$ speedup and a 35.8 $\times$ memory reduction induce only 4.25% relative error for pollution categories. We apply the pipeline to 11 open Chinese corpora and 6 Chinese Common Crawl snapshots from 2021 to 2026. The audit reveals widespread but uneven pollution across open corpora, as well as highly polluted and temporally shifting Chinese web content. We further release a hierarchical Chinese web token dataset with 660k+ token records, each with web context, category, and explanation fields, organized as trees to support review and tracing of pollution.

Problem

• Chinese web pollution
• web-scale corpora
• token-level auditing
• BPE token statistics
• corpus audit

Innovation

• Sampled-BPE
• token-level auditing
• Chinese web pollution
• BPE tokenizer
• web-scale corpora