Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs

2026-06-24

AI Summary

Long-form first-person videos are challenging for multimodal large language models due to input token limitations, often resulting in the loss of temporal and contextual information and thereby hindering fine-grained reasoning. To address this, this work proposes temporally aligned egocentric scene graphs (EgoSGs), which compress videos into compact, structured textual representations that explicitly model objects, attributes, spatial relationships, and their temporal interactions. By preserving critical visual and semantic content within a constrained token budget, EgoSGs bridge the gap between the rich semantics of egocentric videos and the input constraints of current models. This approach represents the first integration of structured, temporally aligned scene graphs into first-person video understanding and achieves state-of-the-art performance on the HD-EPIC VQA benchmark, demonstrating both its effectiveness and generalizability.

Abstract

Existing multi-modal large language models (MLLMs) face significant challenges in processing long video sequences due to strict input token limitations. As a result, current video understanding approaches, especially in egocentric settings characterized by complex dynamics, frequent state changes, and moving cameras, are forced to massively subsample frames. This leads to severe loss of temporal and contextual information, constraining their ability to perform fine-grained video reasoning. In this work, we introduce a framework for egocentric video question answering (VQA) that overcomes these input constraints through Egocentric Scene Graphs (EgoSGs), i.e., temporally grounded, structured representations that capture objects, attributes, spatial relations, and interactions over time. By representing videos as compact, text-based scene graphs, our method preserves the essential visual and temporal information of the original video in a symbolic form that drastically reduces input length while maintaining semantic richness. Crucially, this enables MLLMs to reason efficiently over entire video sequences within their token budget. On HD-EPIC VQA, our method achieves state-of-the-art results, outperforming strong video-based baselines on multiple models and suggesting that structured, temporally grounded representations like EgoSGs can bridge long-form egocentric video understanding and the context limitations of today's MLLMs.

Problem

• egocentric video
• long-form video understanding
• temporal reasoning
• input token limitation
• video question answering

Innovation

• Egocentric Scene Graphs
• Long-form Video Understanding
• Multimodal Large Language Models
• Video Question Answering
• Temporal Grounding