StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videos

2026-08-19

AI Summary

为解决长视频中目标对象不可见时的时空推理问题，提出StateTrace框架，通过构建时空状态记忆来推断对象状态。

Abstract

Existing VLMs have achieved strong performance in video understanding, yet they struggle with long-video spatiotemporal reasoning when target objects become invisible, often mistaking "invisible" for "unknown". We define this challenge as hidden-state spatiotemporal reasoning: inferring object states during prolonged invisible intervals from context interactions. To address this, we propose StateTrace, a novel object-centric framework that endows VideoLLMs with an explicit mechanism for hidden state reasoning in long videos. StateTrace builds a reusable spatiotemporal state memory that organizes object trajectories, inter-object relations, and state-transition events into a structured reasoning substrate. At inference time, it retrieves question-relevant state-evolution trajectories and converts them into compact reasoning cues, enabling the model to explicitly reason about why an object disappears, how its state evolves while invisible, and whether that state should persist at query time. We further build HSR-Bench, a diagnostic benchmark for hidden-state reasoning, containing 1,427 video-QA samples from 1,384 unique videos. Extensive experiments across multiple VideoLLMs show that StateTrace consistently improves performance on both public benchmarks and HSR-Bench (e.g., improving VideoLLaMA3 from 39.6 to 64.2 on HSR-Bench).

Problem

• hidden-state spatiotemporal reasoning
• long videos
• object-centric
• invisible intervals

Innovation

• hidden-state spatiotemporal reasoning
• object-centric framework
• spatiotemporal state memory