GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenes

2026-08-11

AI Summary

Existing 4D scene graphs struggle to model the structured temporal dynamics of human activities, and current activity representations either decouple from persistent 3D scenes or rely on predefined event boundaries and object associations. This work proposes a spatiotemporal memory system that integrates a persistent 4D scene graph with a dual-layer human–object interaction framework and a goal-driven hierarchical event structure to automatically extract, associate, and cluster atomic interactions from RGB-D streams into structured events. Through hierarchical event modeling and a context-aware object association refinement mechanism, the system enables retrospective and inferable activity memory. It achieves performance of 0.71–0.75 on standard benchmarks and scores 0.73 and 0.75 on newly introduced Space2Event and Event2Space query tasks, respectively, approaching the upper bound attainable with ground-truth events and associations.

Abstract

Robots operating in human environments need memories that capture not only what objects exist and where, but also how people use them over time and how individual interactions compose into goal-directed activities. Existing 4D scene graphs preserve object and place histories but omit activity structure, whereas activity representations are either not grounded in persistent 3D scenes or rely on externally provided event boundaries and object associations. We present GESTO (Grounded Event and Spatio-Temporal memOry), a spatio-temporal memory that couples a persistent 4D scene graph with a two-level hierarchy of atomic human--object interactions and goal-driven events. From an RGB-D observation stream, GESTO automatically extracts timestamped interactions, grounds them to persistent scene entities, groups them into events, and uses event context to refine uncertain object associations. A relation-aware tool-calling agent queries the resulting memory for activity-centric spatio-temporal reasoning. We evaluate GESTO on the reproducible text, binary, and time categories of an existing benchmark, together with 40 new Space2Event and Event2Space queries. GESTO achieves scores of 0.71, 0.75, and 0.70 on the standard categories, approaching a method supplied with ground-truth event and object grounding, while substantially outperforming the same reasoning framework when these inputs are removed. It further achieves 0.73 and 0.75 on Space2Event and Event2Space queries. Ablations show that hierarchical event structure and context-aware grounding refinement provide complementary benefits, supporting activity-grounded hierarchical memory for retrospective reasoning in dynamic human environments.

Problem

• spatio-temporal memory
• human-object interaction
• activity reasoning
• 4D scene graph
• event grounding

Innovation

• spatio-temporal memory
• 4D scene graph
• human-object interaction
• hierarchical event structure
• activity-grounded reasoning