Gaze-Informed Proactive AI Assistance for Children's Picture Exploration

2026-07-01

AI Summary

This study investigates how to deliver timely and contextually appropriate proactive AI assistance to children by leveraging their nonverbal behaviors, particularly gaze. To this end, we present Ollie, a novel system that employs eye tracking as an implicit input to model children’s attentional focus and dynamically prompt a large language model (LLM) to generate concise, region-relevant narratives that provide situated guidance. Our approach innovatively integrates attention modeling with LLM-driven content generation to enable adaptive selection of both intervention timing and verbal output. Experimental results demonstrate that gaze-guided assistance significantly prolongs children’s visual engagement with target regions and more effectively promotes exploration of relevant image content compared to random prompting. The system received consistently positive feedback from children, parents, and educators.

Abstract

Proactive assistance with large language models (LLMs) has received growing attention in the human computer interaction (HCI) community. However, most past work on proactive LLMs' assistance has focused on adult users and task-oriented settings, leaving open how such systems could support children, whose interests and needs are often expressed through gaze and other nonverbal behaviors rather than explicit requests. In this study, we focus on two key challenges of proactive assistance in children's picture exploration: when to provide assistance and what assistance to provide based on children's nonverbal behaviors. To address these challenges, we introduce Ollie, a gaze-informed proactive artificial intelligence (AI) assistant that offers short narrative descriptions based on where a child is looking. Ollie uses children's gaze to estimate their attention, identify their current visual focus, and select a related picture region for the LLM to verbally describe. In a within-subject experiment, we compared gaze-informed assistance with random assistance. Results show that gaze-informed assistance kept children's attention on their current focus for a longer period of time, and guided them more effectively to related picture regions. Children, parents, and a participating kindergarten teacher viewed Ollie positively and consider that it better matched children's interests when compared with the random assistance. This work shows the feasibility of using gaze as an implicit input for proactive AI assistance for children and provides design implications for future child-centered AI systems.

Problem

• proactive AI assistance
• children
• gaze
• nonverbal behavior
• picture exploration

Innovation

• gaze-informed assistance
• proactive AI
• children's interaction
• large language models
• nonverbal behavior