VEGAS: Human-Aligned Video Caption Evaluation via Gaze

2026-07-09

AI Summary

Existing video captioning models struggle to capture the attentional focus of individual viewers. To address this limitation, this work proposes VEGAS—a training-free evaluation metric that, for the first time, incorporates eye-tracking data collected during testing into video caption assessment. VEGAS employs a cross-modal information-theoretic approach to quantify the alignment between candidate captions and users’ gaze regions, and leverages rejection sampling to select personalized descriptions that best match human visual attention. Experimental results demonstrate that VEGAS significantly improves the correspondence between generated captions and actual human gaze behavior, while also enhancing downstream video retrieval performance.

Abstract

Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention. We propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a dataset of egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then select captions based on VEGAS via rejection sampling without model retraining. Experiments show that VEGAS-selected captions align significantly better with human focus and improve downstream caption-to-video retrieval, demonstrating the practical utility of incorporating viewer attention during inference.

Problem

• video captioning
• gaze
• attention alignment
• vision-language models
• personalized captioning

Innovation

• gaze-based evaluation
• video captioning
• attention alignment
• training-free metric
• cross-modal information theory