Unknown Word Detection for English as a Second Language (ESL) Learners Using Gaze and Pre-trained Language Models

2025-02-14

AI Summary

This paper addresses the challenge of reading comprehension disruption caused by unknown vocabulary among second-language learners. We propose EyeLingo—the first lightweight, real-time unknown-word detection framework integrating eye-tracking behavior with a pre-trained language model (BERT). Methodologically, we innovatively model multimodal interactions between sequential eye-movement features and textual semantics, enabling low-latency, streaming inference. Our key contribution is the first incorporation of eye-tracking data into probabilistic unknown-word modeling, facilitating dynamic, fine-grained lexical support during natural reading. In a user study with 20 participants, EyeLingo achieves 97.6% detection accuracy and a 71.1% F1-score, significantly improving perceived usefulness and user adoption intention. The system provides a deployable technical pathway for adaptive language learning.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Language and Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

English as a Second Language (ESL) learners often encounter unknown words that hinder their text comprehension. Automatically detecting these words as users read can enable computing systems to provide just-in-time definitions, synonyms, or contextual explanations, thereby helping users learn vocabulary in a natural and seamless manner. This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%. We implemented a real-time reading assistance prototype to show the effectiveness of EyeLingo. The user study shows improvement in willingness to use and usefulness compared to baseline methods.

Problem

• Detect unknown words for ESL learners
• Use gaze and pre-trained models
• Improve reading comprehension and vocabulary learning

Innovation

• Transformer-based machine learning
• Real-time gaze trajectory analysis
• High accuracy unknown word detection