Probing for Reading Times

2026-04-20

AI Summary

This study investigates whether internal representations of language models encode cognitive signals relevant to human reading, with a focus on their ability to predict eye-tracking measures of reading time. Leveraging eye-movement corpora across five languages—English, Greek, Hebrew, Russian, and Turkish—the authors employ regularized linear regression to systematically compare the predictive power of layer-wise model representations against traditional scalar predictors such as surprisal across multiple reading-time metrics. The findings reveal that early layers of language models outperform surprisal in predicting early-stage measures like first fixation and gaze duration, whereas surprisal remains superior for later-stage metrics such as total reading time. Combining both sources yields further performance gains, and the optimal configuration varies by language and metric, highlighting a functional alignment between model depth and distinct phases of human reading processing.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web search models and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.

Problem

• reading times
• eye-tracking
• language models
• cognitive processing
• probing

Innovation

• probing
• reading times
• language models
• eye-tracking
• surprisal