Eye2Recall: Exploring the Design of Enhancing Reminiscence Activities via Eye Tracking-Based LLM-Powered Interaction Experience for Older Adults

2025-08-04

AI Summary

This study addresses the challenges of insufficient natural interaction and weak emotional connection in photo-recall conversations with older adults. We propose a hybrid active interaction paradigm integrating eye-tracking and speech modalities. Methodologically, we develop a large language model (LLM)-based multimodal intelligent reminiscence assistance system that jointly models real-time eye-tracking data—capturing visual attention—and speech input—encoding linguistic intent—to drive personalized, context-aware dialogue generation. Our key contribution is the first deep integration of dual-channel (eye-tracking + speech) active interaction with LLMs, overcoming limitations of unimodal approaches. A user study with older adults demonstrates significant improvements: +32% in conversational fluency, +41% in emotional resonance, and enhanced user engagement. The system effectively supports autonomous narrative construction and promotes active aging.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Photo-based reminiscence has the potential to have a positive impact on older adults' reconnection with their personal history and improve their well-being. Supporting reminiscence in older adults through technological implementations is becoming an increasingly important area of research in the fields of HCI and CSCW. However, the impact of integrating gaze and speech as mixed-initiative interactions in LLM-powered reminiscence conversations remains under-explored. To address this, we conducted expert interviews to understand the challenges that older adults face with LLM-powered, photo-based reminiscence experiences. Based on these design considerations, we developed Eye2Recall, a system that integrates eye tracking for detecting visual interest with natural language interaction to create a mixed-initiative reminiscence experience. We evaluated its effectiveness through a user study involving ten older adults. The results have important implications for the future design of more accessible and empowering reminiscence technologies that better align with older adults' natural interaction patterns and enhance their positive aging.

Problem

• Enhancing reminiscence for older adults using eye-tracking and LLM
• Exploring mixed-initiative interactions in photo-based reminiscence activities
• Improving accessibility of reminiscence technologies for natural aging patterns

Innovation

• Eye tracking for visual interest detection
• LLM-powered natural language interaction
• Mixed-initiative reminiscence experience design