Gaze patterns predict preference and confidence in pairwise AI image evaluation

2026-03-25

AI Summary

This work addresses a critical limitation in existing preference learning methods, which rely on pairwise human judgments while overlooking the underlying cognitive mechanisms—particularly in the evaluation of AI-generated images. Through an eye-tracking experiment, the study reveals for the first time a gaze cascade effect during pairwise image assessment and demonstrates that this effect can simultaneously predict both preference choices and decision confidence. Leveraging oculomotor features—including fixation duration, fixation count, regressions, and image switches—the authors develop a model that achieves 68% accuracy in binary preference prediction and 66% accuracy in distinguishing high versus low decision confidence. These findings provide a quantifiable, implicit physiological basis for assessing the quality of human preference annotations.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Humans and AI: Learning Human Values and Preferences
• Knowledge Representation and Reasoning: Preferences

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Preference learning methods, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), rely on pairwise human judgments, yet little is known about the cognitive processes underlying these judgments. We investigate whether eye-tracking can reveal preference formation during pairwise AI-generated image evaluation. Thirty participants completed 1,800 trials while their gaze was recorded. We replicated the gaze cascade effect, with gaze shifting toward chosen images approximately one second before the decision. Cascade dynamics were consistent across confidence levels. Gaze features predicted binary choice (68% accuracy), with chosen images receiving more dwell time, fixations, and revisits. Gaze transitions distinguished high-confidence from uncertain decisions (66% accuracy), with low-confidence trials showing more image switches per second. These results show that gaze patterns predict both choice and confidence in pairwise image evaluations, suggesting that eye-tracking provides implicit signals relevant to the quality of preference annotations.

Problem

• preference learning
• gaze patterns
• pairwise evaluation
• decision confidence
• human feedback

Innovation

• eye-tracking
• preference learning
• gaze cascade
• confidence prediction
• AI image evaluation