To model human linguistic prediction, make LLMs less superhuman

2025-10-01

AI Summary

This paper addresses the problem that large language models (LLMs) significantly outperform humans in linguistic prediction tasks, thereby reducing cognitive fidelity—i.e., alignment with human behavioral and neural responses. To mitigate this “superhuman” divergence, we propose a *de-superhumanization* modeling paradigm: systematically attenuating LLMs’ long-term factual memory and short-term contextual memory to better match human lexical frequency sensitivity and reading difficulty responses. Methodologically, we analyze memory-related biases in LLMs using cognitive neuroscience data (e.g., eye-tracking, ERP), and design controllable memory decay and context truncation strategies. Our key contributions are: (1) identifying the paradoxical mechanism wherein improved predictive accuracy degrades human behavioral alignment; (2) establishing the first cognitive alignment framework explicitly grounded in human memory constraints as a prior; and (3) highlighting the absence of cognition-grounded metrics in current evaluation benchmarks—thereby providing theoretical foundations and empirical pathways toward interpretable, verifiable, human-level language processing models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

When people listen to or read a sentence, they actively make predictions about upcoming words: words that are less predictable are generally read more slowly than predictable ones. The success of large language models (LLMs), which, like humans, make predictions about upcoming words, has motivated exploring the use of these models as cognitive models of human linguistic prediction. Surprisingly, in the last few years, as language models have become better at predicting the next word, their ability to predict human reading behavior has declined. This is because LLMs are able to predict upcoming words much better than people can, leading them to predict lower processing difficulty in reading than observed in human experiments; in other words, mainstream LLMs are 'superhuman' as models of language comprehension. In this position paper, we argue that LLMs' superhumanness is primarily driven by two factors: compared to humans, LLMs have much stronger long-term memory for facts and training examples, and they have much better short-term memory for previous words in the text. We advocate for creating models that have human-like long-term and short-term memory, and outline some possible directions for achieving this goal. Finally, we argue that currently available human data is insufficient to measure progress towards this goal, and outline human experiments that can address this gap.

Problem

• LLMs predict words better than humans, reducing reading behavior accuracy
• Superhuman performance stems from superior long-term and short-term memory
• Need models with human-like memory to predict linguistic processing

Innovation

• Reduce LLMs' long-term memory capacity
• Limit LLMs' short-term memory capabilities
• Create human-like memory architecture models