Reading the Eyes in VR: Multimodal Modeling of Social Intelligence

2026-07-10

AI Summary

This study addresses limitations of the traditional Reading the Mind in the Eyes Test (RMET)—notably inconsistent viewpoints and lack of process-oriented feedback—that hinder precise assessment of social intelligence. For the first time, the authors implement a standardized RMET task in virtual reality (VR), developing a cross-platform (VR/desktop) experimental system in Unity that synchronously captures eye-tracking, electroencephalography (EEG), behavioral responses, and subjective questionnaire data, while incorporating an immediate feedback mechanism. Results demonstrate that immediate feedback significantly prolongs gaze duration and enhances EEG markers associated with cognitive engagement. Although VR did not improve objective task performance, it received higher usability ratings, supporting its viability as a controlled, process-sensitive environment for assessing social cognition.

Abstract

Social intelligence, the ability to interpret others' emotions, beliefs, and intentions, is often assessed with the Reading the Mind in the Eyes Test (RMET), in which participants infer mental states from images of the eye region. Yet RMET is typically presented on paper or desktop displays, where viewing geometry can vary across participants, and it rarely includes immediate feedback. We investigated whether presentation medium and brief trial-level feedback influence RMET behavior. We implemented RMET in Unity for both desktop and Virtual Reality (VR), using VR to hold stimulus distance and field of view constant without changing the items. We conducted a 2x2 mixed study with 20 participants, with device (VR vs. desktop) manipulated between subjects and feedback (immediate correctness cue vs. none) manipulated within subjects. Eye-tracking and EEG data were recorded and synchronized with behavioral logs. We analyzed fixation-based gaze measures, EEG signals, response time, accuracy, and subjective measures. Immediate feedback was associated with longer fixation durations and higher EEG-based engagement, while no significant differences were observed in task completion time or total correct answers. Presentation medium did not produce reliable differences in the objective measures, but VR received higher usability ratings and was also rated as more effortful. These results provide initial evidence that RMET can be studied as a process-aware assessment task in controlled VR and desktop settings.

Problem

• Reading the Mind in the Eyes Test
• social intelligence
• virtual reality
• eye-tracking
• feedback

Innovation

• Virtual Reality
• Multimodal Modeling
• Eye-tracking
• EEG
• Social Intelligence Assessment