Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

2026-01-14

AI Summary

This study investigates why language model (LM) prediction probabilities outperform traditional cloze task measures in capturing word predictability and whether this advantage stems from cognitively plausible mechanisms. By systematically comparing LM surprisal with human cloze probabilities through psycholinguistic experiments and computational linguistic analyses, the work provides the first empirical validation of three core hypotheses: LMs offer higher resolution, greater semantic discriminability, and more accurate modeling of low-frequency words. The findings demonstrate that LMs significantly surpass conventional cloze methods across these dimensions, offering novel insights into human language prediction mechanisms and fostering deeper integration between psycholinguistics and computational linguistics.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

How predictable a word is can be quantified in two ways: using human responses to the cloze task or using probabilities from language models (LMs).When used as predictors of processing effort, LM probabilities outperform probabilities derived from cloze data. However, it is important to establish that LM probabilities do so for the right reasons, since different predictors can lead to different scientific conclusions about the role of prediction in language comprehension. We present evidence for three hypotheses about the advantage of LM probabilities: not suffering from low resolution, distinguishing semantically similar words, and accurately assigning probabilities to low-frequency words. These results call for efforts to improve the resolution of cloze studies, coupled with experiments on whether human-like prediction is also as sensitive to the fine-grained distinctions made by LM probabilities.

Problem

• language model surprisal
• cloze surprisal
• prediction in language comprehension
• processing effort
• human-like prediction

Innovation

• language model surprisal
• cloze task
• prediction resolution
• semantic discrimination
• low-frequency words