R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video

2026-08-11

AI Summary

Answering object-centric questions—such as “Where was an item moved?” or “When did its state change?”—in long-duration first-person videos remains challenging due to the inability of existing methods to model persistent object identities and structured spatial changes. This work proposes a relative 4D scene graph memory that encodes videos into compact memory entries indexed by time, location, persistent objects, relative transformations from stable anchors, and local interaction contexts. By decoupling stable anchors from dynamic objects and representing state evolution through relative transformations, the approach maintains cross-frame object identity without requiring a globally aligned world model and enables efficient retrieval. Built upon RGB-only promptable segmentation, temporal propagation, and relative 3D lifting, the end-to-end retrievable system outperforms EgoRAG-Text by 6.7 points on 255 object-related questions in the EgoLifeQA dataset using question-only retrieval, with a remarkable 12.5-point gain on “when”-type questions.

Abstract

Long-horizon egocentric video is a rich substrate for wearable AI assistants, but object-centric questions such as where an item was moved, when it last changed state, or why it was relocated remain difficult because caption- and transcript-based memories rarely preserve persistent object identity or structured spatial change. Existing long-video QA methods mainly emphasize temporal grounding and clip retrieval, while prior 3D scene-graph methods typically assume stronger geometry than free-motion wearable RGB video provides, including point clouds, RGB-D input, posed views, sparse reconstruction, or reconstructed scenes. R4DSG introduces a relative 4D scene graph memory for long egocentric video. Instead of storing raw graph sequences, R4DSG converts video into compact queryable memory entries indexed by time, place, persistent objects, anchor-relative change, and local interaction context. The main idea is to separate stable anchors from dynamic objects, maintain persistent object identity across frames, and represent object state through anchor-relative transitions rather than a globally aligned world model. Built on recent RGB-only advances in promptable video segmentation, temporal propagation, and relative 3D lifting, the method produces a retrieval-ready memory directly usable for long-horizon question answering. Evaluation on a 255-question object-related subset from EgoLifeQA shows, under question-only retrieval, a 6.7-point overall gain over EgoRAG-Text and a 12.5-point gain on when questions, which highlights the value of temporally organized object memory. These results position relative 4D scene graphs as a practical memory substrate for wearable assistants, AR systems, and embodied multimedia agents. GitHub Page: https://dualtransparency.github.io/R4DSG/.

Problem

• egocentric video
• object-centric question answering
• long-horizon QA
• scene graph
• object identity

Innovation

• relative 4D scene graph
• object-centric QA
• egocentric video
• persistent object identity
• anchor-relative representation