Decoding Reading Goals from Eye Movements

2024-10-28

arXiv.org

AI Summary

This work investigates whether readers’ reading goals—information seeking versus comprehension reading—can be decoded in real time from their eye movement trajectories. To address this, we introduce the first large-scale, annotated eye-tracking dataset and propose a novel Transformer architecture that jointly models scanpath representations and contextualized semantic features from pretrained language models; we further incorporate a mixed-effects model to disentangle text difficulty from inter-subject variability. Methodologically, our approach innovatively integrates sequential visual behavior with linguistic semantics and employs ensemble learning to enhance robustness. Experiments demonstrate that the model achieves high-accuracy goal classification early in reading—well before completion—and identify key determinants of classification difficulty, including textual features (e.g., information density, structural complexity) and individual differences. These findings substantially advance the understanding of oculomotor correlates underlying distinct cognitive mechanisms in the two reading modes.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Readers can have different goals with respect to the text that they are reading. Can these goals be decoded from their eye movements over the text? In this work, we examine for the first time whether it is possible to distinguish between two types of common reading goals: information seeking and ordinary reading for comprehension. Using large-scale eye tracking data, we address this task with a wide range of models that cover different architectural and data representation strategies, and further introduce a new model ensemble. We find that transformer-based models with scanpath representations coupled with language modeling solve it most successfully, and that accurate predictions can be made in real time, long before the participant finished reading the text. We further introduce a new method for model performance analysis based on mixed effect modeling. Combining this method with rich textual annotations reveals key properties of textual items and participants that contribute to the difficulty of the task, and improves our understanding of the variability in eye movement patterns across the two reading regimes.

Problem

• Decoding reading goals from eye movements
• Distinguishing information seeking vs comprehension
• Real-time prediction using transformer-based models

Innovation

• Transformer-based models with scanpaths
• Real-time prediction using eye movements
• Mixed effect modeling for analysis