The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

2026-05-07

AI Summary

Surprisal from language models is commonly employed as a proxy for metaphorical novelty, yet it is highly confounded with word frequency, potentially leading to attribution bias. This study systematically investigates the relationship between surprisal and metaphorical novelty by leveraging eight model sizes of Pythia, 154 training checkpoints, and two distinct word frequency measures, evaluated through regression analyses. The findings reveal that word frequency consistently emerges as a stronger predictor of novelty than surprisal. Moreover, the association between surprisal and novelty peaks early in training and rapidly diminishes thereafter, challenging prevailing assumptions about optimal surprisal mechanisms in language models and suggesting that prior results may have erroneously attributed frequency effects to contextual predictability.

Abstract

Language-model (LM) surprisal is widely used as a proxy for contextual predictability and has been reported to correlate with metaphor novelty judgments. However, surprisal is tightly intertwined with lexical frequency. We explore this interaction on metaphor novelty ratings using two different word frequency measures. We analyse surprisal estimates from eight Pythia model sizes and 154 training checkpoints. Across settings, word frequency is a stronger predictor of metaphor novelty than surprisal. Across training stages, the surprisal--novelty association peaks at an early stage and then falls again, mirroring a similarly timed increase in the surprisal--frequency association. These results suggest that the often-reported optimal LM surprisal settings may incorrectly associate contextual predictability with metaphor novelty and processing difficulty, whereas lexical frequency may be the major underlying factor.

Problem

• language model surprisal
• metaphor novelty
• lexical frequency
• frequency confound
• contextual predictability

Innovation

• surprisal
• lexical frequency
• metaphor novelty
• language models
• frequency confound