When Is Noise Response Universal? Tokenization as the Hidden Variable in Language Models

2026-08-26

AI Summary

研究解决了文本模型对噪声敏感的问题，通过分析不同噪声规模下的性能降级，发现训练目标而非架构是关键因素，并提出一种无需噪声评估即可预测模型鲁棒性的方法。

Abstract

The performance of textual neural models often degrades when their inputs are corrupted by noise such as typos, OCR errors, or dropped words. We study the degradation rate across neural models, both sentence embeddings and decoder-only LLMs, and find that how consistent it is depends on the scale of the noise: under word-level noise, models with very different architectures decline along nearly the same curve, while under character-level noise they separate. We further identify the determining factor to be the training objective, not the architecture: eight encoders spanning six pretraining paradigms are scattered initially, and collapse onto a common curve after a short contrastive training recipe. We trace the word/character split to tokenization: a single character edit forces the tokenizer to re-segment the surrounding word, disturbing the token sequence far more than dropping a whole word does. This finding and its underlying mechanism provide a practical means to predict a model's robustness to noise without any noisy evaluation, and to install robustness at a chosen noise scale through noise-augmented training.

Problem

• noise
• language models
• tokenization

Innovation

• Noise Response
• Tokenization
• Contrastive Training
• Robustness to Noise