LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension

2026-07-02

AI Summary

This work addresses the limitations of existing referring expression comprehension benchmarks, which are confined to short video clips and struggle with unedited egocentric videos spanning up to 45 minutes that exhibit sparse targets and complex activities. To bridge this gap, the authors introduce a new benchmark tailored for long-form egocentric video understanding, built upon the Ego4D dataset and featuring fine-grained natural language queries paired with precise spatiotemporal annotations. The task requires models to jointly localize the spatiotemporal extent of entities referred to in language within intricate human-centric activities. Experimental results demonstrate a significant performance drop among state-of-the-art models on this benchmark, underscoring the substantial challenges posed by long-duration egocentric video understanding and revealing critical limitations of current vision-language models in handling extended temporal contexts.

Abstract

Egocentric videos capture rich and diverse human-object interactions and have emerged as a fundamental resource for understanding human activities related to objects. In this context, Video Referring Expression Comprehension (Video REC), the task of localizing the temporal and spatial extent of a referred object in video frames given a natural language query, plays a key role in linking textual descriptions to observed objects in untrimmed egocentric recordings. However, existing egocentric Video REC benchmarks primarily focus on short video clips, where some target object appears densely within frames. Such settings do not reflect real-world egocentric recordings, which are long-form, untrimmed, and characterized by sparse object occurrences and complex activity transitions. To address this limitation, we introduce LongEgoRefer, a novel and challenging benchmark constructed from long-form videos in the Ego4D dataset. LongEgoRefer contains 1,498 referring expressions with an average video duration of 45 minutes. The benchmark exhibits extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives. Consequently, it defines a demanding spatio-temporal grounding problem that requires models to identify both when an event occurs and where the referred object appears within extended video sequences. We evaluate existing Video REC approaches, including training-free baselines based on vision-language models combined with Grounded SAM2. Extensive experiments show that even advanced baselines and current state-of-the-art models struggle significantly on LongEgoRefer. These results highlight the intrinsic difficulty of long-form egocentric spatio-temporal grounding and emphasize the need for more robust video understanding models.

Problem

• egocentric video
• referring expression comprehension
• long-form video
• spatio-temporal grounding
• object localization

Innovation

• Long-form egocentric video
• Video Referring Expression Comprehension
• Spatio-temporal grounding
• Target sparsity
• Ego4D