STORM: Internalized Modeling for Spatial-Temporal Reasoning in Video-Language Models

2026-05-25

AI Summary

This work proposes STORMS, a novel framework that internalizes spatiotemporal reasoning within video-language models by representing it as a bounded continuous trajectory in latent space, eliminating the need for external methods such as textual reasoning chains, keyframe selection, or tool invocation during inference. By avoiding on-the-fly video generation, frame interpolation, or repeated encoding of visual evidence, STORMS significantly reduces latency and engineering complexity. The approach employs a two-stage training strategy: first aligning latent tokens with dynamic visual representations using generated videos, then reinforcing end-to-end reasoning capabilities through answer-only supervision. Evaluated on VideoMME, MVBench, TempCompass, and MMVU benchmarks, STORMS achieves substantial gains in accuracy while markedly lowering inference overhead, enabling efficient and fully integrated spatiotemporal understanding.

Abstract

Many video reasoning tasks require tracking motion, temporal order, and evolving visual states across frames. Existing methods built on large vision-language models (LVLMs) often address this challenge by externalizing reasoning through textual chain-of-thought (CoT), keyframe selection, repeated frame reinsertion, or external tool use. While effective, such pipelines increase inference-time latency and engineering complexity, and they force temporal-visual evidence to be serialized into text or repeatedly re-encoded from frames. Inspired by the intuition that visual reasoning can occur implicitly before verbalization, we propose STORMS (Spatial-Temporal reasOning via inteRnalized Modeling), a two-stage framework that teaches LVLMs to reason through bounded continuous latent trajectories instead of explicit textual CoT. In Stage I, STORMS aligns latent tokens with thought-video representations derived from generated videos, grounding the latent states in dynamic visual evidence. In Stage II, the model is further trained with answer-only supervision, encouraging the reasoning process to be internalized without step-by-step annotations. Generated thought videos are used only during training; at inference, STORMS performs a bounded latent rollout without regenerating videos, reinserting frames, or invoking external visual tools. Experiments on VideoMME, MVBench, TempCompass, and MMVU show that STORMS improves video reasoning accuracy while substantially reducing inference overhead compared with tool or video-generation-based reasoning pipelines.

Problem

• video-language models
• spatial-temporal reasoning
• internalized modeling
• visual reasoning
• inference latency

Innovation

• internalized reasoning
• spatial-temporal modeling
• latent trajectory
• video-language models
• thought video