SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding

2026-07-04

AI Summary

Existing approaches typically model human actions and gaze in isolation, overlooking their intrinsic coupling in behavior understanding. This work proposes SAGE, a unified framework that, for the first time, jointly models the recognition and future prediction of human-object interaction (HOI) and gaze within an end-to-end trainable architecture applicable to both egocentric and exocentric settings. Built upon Transformers, SAGE integrates gaze cues into spatiotemporal attention mechanisms to enable joint reasoning over current and future action-gaze dynamics. We also introduce Exo-Cook, the first dataset with synchronized HOI and gaze annotations for exocentric videos. Experiments demonstrate that SAGE achieves performance on par with or superior to state-of-the-art methods specialized for individual tasks across VidHOI, EGTEA Gaze+, and Exo-Cook benchmarks.

Abstract

Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook to facilitate further research in this domain. Experiments on three benchmark datasets: VidHOI, EGTEA Gaze+, and Exo-Cook show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.

Problem

• Human Object Interaction
• Gaze Prediction
• Action Anticipation
• Behavior Understanding
• Spatiotemporal Modeling

Innovation

• Synchronized Action-Gaze Recognition
• Human Object Interaction (HOI)
• Gaze Anticipation
• Transformer-based Architecture
• Exocentric Benchmark