Energy-Based Transformers as Predictors of Reading Difficulty

2026-06-22

AI Summary

This study addresses the challenge of accurately predicting human reading difficulty by integrating disparate cognitive load metrics into a unified framework. The authors propose an energy-based Transformer model, leveraging its theoretical connection to associative memory systems such as Hopfield networks to construct a single, coherent energy measure. This work introduces energy-based Transformers to computational psycholinguistics for the first time. The proposed metric simultaneously captures the predictive power of both surprisal and attention entropy, significantly outperforming traditional surprisal measures across multiple eye-tracking corpora. Furthermore, it successfully replicates the well-known subject–object asymmetry in relative clause processing, demonstrating strong generalization capabilities and validating its cognitive plausibility.

Abstract

Transformer language models have become established tools for modeling human sentence processing, with measures such as surprisal and attention entropy serving as effective predictors of reading difficulty that together capture complementary aspects of processing load. Here, we explore a related class of transformer models: energy-based transformers, which provide a principled formal link to associative memory models, bringing processing research into direct contact with the broader literature on Hopfield networks and dense associative memory. To our knowledge, this is the first exploration of an energy-based transformer measure in computational psycholinguistics. Across reading-time corpora (Natural Stories, UCL eye-tracking, UCL self-paced reading), the energy measure is a robust predictor of reading times, providing significant fit beyond surprisal in all three. In a controlled experiment on relative clause processing, energy at a single layer captures the well-known object/subject asymmetry. We find evidence that it subsumes effects attributable to both attention entropy and surprisal, suggesting that energy may serve as a single unified predictor where multiple complementary measures have previously been required.

Problem

• reading difficulty
• energy-based transformers
• surprisal
• attention entropy
• psycholinguistics

Innovation

• energy-based transformers
• reading difficulty prediction
• associative memory
• unified processing measure
• computational psycholinguistics