A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes

2024-08-02

arXiv.org

AI Summary

This study investigates how uncertainty and semantic meaning jointly guide human eye movements in dynamic scenes. We propose the first closed-loop computational model integrating boundary uncertainty estimation with semantic object segmentation: a Bayesian filter recursively models object boundaries and their uncertainties, which serve as novel oculomotor control signals for active visual exploration. Key contributions include: (1) revealing the critical regulatory role of boundary uncertainty in the exploration–exploitation trade-off; and (2) demonstrating that semantic objects constitute the fundamental units of attention, with the model implicitly reproducing high-level oculomotor phenomena such as inhibition-of-return delays. Evaluated on real-world dynamic scene datasets, the model faithfully replicates human free-viewing behavior—quantitatively matching fixation durations, saccade amplitude distributions, and the balanced pattern of object detection, inspection, and revisiting. Moreover, it generalizes to higher-order scanpath statistics not used during model fitting.

Technology Category

• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The objects we perceive guide our eye movements when observing real-world dynamic scenes. Yet, gaze shifts and selective attention are critical for perceiving details and refining object boundaries. Object segmentation and gaze behavior are, however, typically treated as two independent processes. Here, we present a computational model that simulates these processes in an interconnected manner and allows for hypothesis-driven investigations of distinct attentional mechanisms. Drawing on an information processing pattern from robotics, we use a Bayesian filter to recursively segment the scene, which also provides an uncertainty estimate for the object boundaries that we use to guide active scene exploration. We demonstrate that this model closely resembles observers' free viewing behavior on a dataset of dynamic real-world scenes, measured by scanpath statistics, including foveation duration and saccade amplitude distributions used for parameter fitting and higher-level statistics not used for fitting. These include how object detections, inspections, and returns are balanced and a delay of returning saccades without an explicit implementation of such temporal inhibition of return. Extensive simulations and ablation studies show that uncertainty promotes balanced exploration and that semantic object cues are crucial to forming the perceptual units used in object-based attention. Moreover, we show how our model's modular design allows for extensions, such as incorporating saccadic momentum or pre-saccadic attention, to further align its output with human scanpaths.

Problem

• Visual Guidance
• Uncertainty
• Attention Allocation

Innovation

• Bayesian filters
• unified approach
• human visual behavior