Integrating large language models and active inference to understand eye movements in reading and dyslexia

📅 2023-08-09
🏛️ arXiv.org
📈 Citations: 0
Influential: 0
📄 PDF
🤖 AI Summary
This study addresses the oculomotor abnormalities underlying developmental dyslexia by proposing a hierarchical active inference computational model. Methodologically, it integrates large language models’ (LLMs) semantic prediction capability with the perception–action closed loop of active inference, constructing a cognitively interpretable generative model that quantitatively simulates core deficits—such as fragmented saccades—via modulation of prior strength to instantiate “weak priors.” Results demonstrate successful replication of typical reading eye-movement patterns and dyslexic characteristics—including short-amplitude, high-frequency saccades—thereby empirically supporting the dual-route reading theory. Crucially, the model provides the first intervention-targetable computational characterization of predictive processing deficits in dyslexia, establishing both theoretical grounding and a mechanistic modeling framework for mechanism-driven reading interventions. (149 words)
📝 Abstract
We present a novel computational model employing hierarchical active inference to simulate reading and eye movements. The model characterizes linguistic processing as inference over a hierarchical generative model, facilitating predictions and inferences at various levels of granularity, from syllables to sentences. Our approach combines the strengths of large language models for realistic textual predictions and active inference for guiding eye movements to informative textual information, enabling the testing of predictions. The model exhibits proficiency in reading both known and unknown words and sentences, adhering to the distinction between lexical and nonlexical routes in dual-route theories of reading. Notably, our model permits the exploration of maladaptive inference effects on eye movements during reading, such as in dyslexia. To simulate this condition, we attenuate the contribution of priors during the reading process, leading to incorrect inferences and a more fragmented reading style, characterized by a greater number of shorter saccades. This alignment with empirical findings regarding eye movements in dyslexic individuals highlights the model's potential to aid in understanding the cognitive processes underlying reading and eye movements, as well as how reading deficits associated with dyslexia may emerge from maladaptive predictive processing. In summary, our model represents a significant advancement in comprehending the intricate cognitive processes involved in reading and eye movements, with potential implications for understanding and addressing dyslexia through the simulation of maladaptive inference. It may offer valuable insights into this condition and contribute to the development of more effective interventions for treatment.
Problem

Research questions and friction points this paper is trying to address.

Model integrates language models and active inference for reading simulation
Examines maladaptive predictive processing in dyslexia-related reading deficits
Tests cognitive processes underlying eye movements during reading
Innovation

Methods, ideas, or system contributions that make the work stand out.

Hierarchical active inference simulates reading processes
Combines large language models with active inference
Attenuating priors mimics dyslexic reading patterns
🔎 Similar Papers
No similar papers found.
F
Francesco Donnarumma
Institute of Cognitive Sciences and Technologies - National Research Council, Via San Martino della Battaglia 44, Rome, 00185, Italy
M
Mirco Frosolone
Institute of Cognitive Sciences and Technologies - National Research Council, Via San Martino della Battaglia 44, Rome, 00185, Italy
Giovanni Pezzulo
Giovanni Pezzulo
National Research Council of Italy, Rome
Embodied CognitionCognitive ScienceCognitive RoboticsGoal-directed BehaviorActive Inference