N-gram-like Language Models Predict Reading Time Best

2026-03-10

AI Summary

Although advanced language models excel at next-word prediction, they perform markedly worse than simple n-gram models in predicting human reading times. This study systematically evaluates the correlation between reading times and model outputs by integrating eye-tracking data, neural language models, and n-gram probabilities. The findings reveal that n-gram–based models significantly outperform complex semantic models such as Transformers in predicting reading times for natural text, suggesting that human reading behavior relies more heavily on local statistical regularities than on deep semantic processing. These results challenge the prevailing assumption that large-scale language models universally capture human-like cognitive processes and offer a new perspective on computational modeling of language comprehension and reading behavior.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Recent work has found that contemporary language models such as transformers can become so good at next-word prediction that the probabilities they calculate become worse for predicting reading time. In this paper, we propose that this can be explained by reading time being sensitive to simple n-gram statistics rather than the more complex statistics learned by state-of-the-art transformer language models. We demonstrate that the neural language models whose predictions are most correlated with n-gram probability are also those that calculate probabilities that are the most correlated with eye-tracking-based metrics of reading time on naturalistic text.

Problem

• reading time
• language models
• n-gram
• transformer
• eye-tracking

Innovation

• n-gram
• reading time prediction
• transformer language models
• eye-tracking
• cognitive modeling