Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs

2026-07-02

AI Summary

Existing approaches struggle to explicitly model the dynamic scene changes induced by human–environment interactions, limiting interpretable and controllable scene evolution reasoning. This work introduces, for the first time, spatiotemporal scene graph sequences to explicitly represent activity-driven scene state evolution in first-person videos, and proposes GLEN, a graph neural network that jointly models the temporal evolution of textual descriptions and graph structures through an action–scene graph alignment mechanism. We formulate a novel task—activity-driven graph edit forecasting (A-GEF)—enabling structured and editable dynamic reasoning. Evaluated on multiple benchmarks including EgoMCQ, EgoCVR, EXPLORE-Bench, and A-GEF, our method significantly outperforms raw video baselines and matches or even surpasses multimodal large language models in long-horizon reasoning tasks.

Abstract

Understanding human behavior while interacting with the surrounding world is crucial for many applications of embodied AI. First-person videos are particularly informative for this problem, as they well capture how activities reshape the scene over time. However, existing approaches often rely on implicit visual or language-aligned representations, disregarding structured reasoning over the scene dynamic. We argue that explicit, compositional and editable representations of human-environment interactions can play a crucial role for rich grounded activity understanding. To this end, we introduce SG-Ego, a large scale annotation set extending Ego4D with spatio-temporal scene graphs, where relations triplets are consolidated over time into explicit time-evolving descriptions of the scene state. To reason over this representation, we propose GLEN, a graph-based model that operates over scene graph sequences to both align them with textual actions and model their temporal evolution. In addition, we formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations conditioned on ongoing actions, enabling explicit reasoning about how scenes change over time. We validate our approach across multiple downstream tasks, spanning retrieval benchmarks as EgoMCQ and EgoCVR, as well as long-horizon reasoning benchmarks as EXPLORE-Bench and the newly introduced A-GEF. GLEN achieves strong results compared to raw video baselines and it excels in reasoning settings, typically addressed only with MLLMs, while enabling controllable and structured predictions of scene dynamics driven by human activities. We believe our results establish spatio-temporal scene graphs, together with models that reason over them, as strong compositional and interpretable representations for video understanding and potentially beyond.

Problem

• scene graphs
• human activities
• scene dynamics
• first-person videos
• structured reasoning

Innovation

• spatio-temporal scene graphs
• structured reasoning
• graph-based modeling
• activity-driven forecasting
• compositional representation