Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?

2024-05-28

AI Summary

Existing egocentric video-language models (EgoVLMs) exhibit significant limitations in fine-grained semantic understanding of hand-object interactions (HOIs), particularly in discriminating subtle variations in verbs or nouns. We identify two fundamental bottlenecks: insufficient discriminability of negative samples in text-video alignment and weak object-centric feature modeling. To address these, we introduce EgoHOIBench—the first benchmark specifically designed for fine-grained HOI understanding—and propose EgoNCE++, an asymmetric contrastive learning objective. EgoNCE++ enhances textual discriminability via large language model–generated semantically relevant negatives and HOI vocabulary substitution, while simultaneously enforcing object-centric clustering through object-aware video feature space modeling. Extensive experiments across multiple EgoVLM backbones demonstrate consistent and substantial improvements in multi-instance cross-modal retrieval, action recognition, and temporal localization, validating both effectiveness and generalizability.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Egocentric video-language pretraining is a crucial step in advancing the understanding of hand-object interactions in first-person scenarios. Despite successes on existing testbeds, we find that current EgoVLMs can be easily misled by simple modifications, such as changing the verbs or nouns in interaction descriptions, with models struggling to distinguish between these changes. This raises the question: Do EgoVLMs truly understand hand-object interactions? To address this question, we introduce a benchmark called EgoHOIBench, revealing the performance limitation of current egocentric models when confronted with such challenges. We attribute this performance gap to insufficient fine-grained supervision and the greater difficulty EgoVLMs experience in recognizing verbs compared to nouns. To tackle these issues, we propose a novel asymmetric contrastive objective named EgoNCE++. For the video-to-text objective, we enhance text supervision by generating negative captions using large language models or leveraging pretrained vocabulary for HOI-related word substitutions. For the text-to-video objective, we focus on preserving an object-centric feature space that clusters video representations based on shared nouns. Extensive experiments demonstrate that EgoNCE++ significantly enhances EgoHOI understanding, leading to improved performance across various EgoVLMs in tasks such as multi-instance retrieval, action recognition, and temporal understanding. Our code is available at https://github.com/xuboshen/EgoNCEpp.

Problem

• Understanding hand-object interactions in egocentric videos
• EgoVLMs struggle with verb and noun modifications
• Insufficient fine-grained supervision in current EgoVLMs

Innovation

• Introduces EgoHOIBench for model evaluation
• Proposes EgoNCE++ for enhanced supervision
• Focuses on object-centric video clustering