Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?

2026-07-09

AI Summary

This work addresses the susceptibility of existing video-language models to spurious correlations in human-object interaction (HOI) understanding, which hinders their ability to capture the true dynamics between hand manipulations and object state changes. To overcome this limitation, the authors propose a novel learning paradigm that explicitly models central representations of hands and objects through hand-object masking training and an HOI dynamics-aware decoder. Auxiliary position and semantic prediction tasks are introduced to enhance sensitivity to critical cues. The study further introduces CI-HOI, the first evaluation framework for disentangling hand and object clues, along with the DEHOI benchmark. Experiments demonstrate that the proposed method significantly outperforms current models on DEHOI, action recognition, object state recognition, and robotic manipulation tasks, thereby improving the robustness and generalization of HOI understanding.

Abstract

Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations. However, existing video-language models often fall into shortcuts by relying on spurious correlations among hands, objects, or environmental context, rather than reasoning from the appearance and dynamics of hands and objects themselves. To address this limitation, we propose a new learning paradigm that combines (i) hand-object masked training, which enables robust reasoning from partial hand or object observations, and (ii) an HOI-dynamics-aware decoder that explicitly learns hand- and object-centric embeddings through auxiliary predictions of their locations and semantics, enhancing sensitivity to both cues. To systematically evaluate such cue-specific reasoning, we introduce Cue-Isolated HOI (CI-HOI), a new evaluation that assesses models' ability to predict actions from hand- and object-related cues independently. To enable CI-HOI, we curate the DEHOI testbed, which separates hand- and object-related observations for disentangled HOI evaluation through inpainting. Using DEHOI, we demonstrate both quantitatively and qualitatively that our training strategy exploits hand- and object-centric information more effectively than existing models. Our approach improves over existing models on DEHOI, standard action recognition, object state recognition, and even robot manipulation action recognition, leading to more robust HOI understanding.

Problem

• hand-object interaction
• video-language models
• spurious correlations
• cue-specific reasoning
• object-centric cues

Innovation

• hand-object interaction
• masked training
• cue-isolated evaluation
• HOI-dynamics-aware decoder
• disentangled representation