SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video

2026-07-02

AI Summary

This work addresses key challenges in pre-trained vision-language models, including spatial ambiguity arising from 2D observations, scarcity of data for 3D spatial understanding, insufficient video input information, and weak reasoning constraints. To tackle these issues, the authors propose SpaceEra++, a unified framework that constructs compact yet semantically and spatially balanced scene representations through a novel ScenePick frame sampling strategy. SpaceEra++ further introduces the SpaceAlign mechanism, which enforces pairwise object constraints by jointly leveraging absolute coordinates and relative spatial relationships to enhance 3D reasoning. Combined with multi-stage training optimization and 3D-aware prompt engineering, the framework significantly outperforms strong baselines across multiple benchmarks. Ablation studies confirm the effectiveness of each component, establishing SpaceEra++ as a new paradigm for 3D vision-language research.

Abstract

Visual-spatial understanding, defined as the ability to infer object relationships and scene layouts from visual inputs, is fundamental to downstream tasks such as robotic navigation and embodied interaction. However, pre-trained vision-language models (VLMs) remain constrained by spatial uncertainty stemming from inherently 2D observations and by the scarcity of data for 3D spatial understanding. To address these limitations, we proposed a novel framework, SpaceEra, in the NeurIPS 2025 Spotlight paper. Although it achieved significant performance gains, we further observed that its effectiveness is hindered by insufficient input from scanning videos and weak reasoning constraints. To tackle these newly emerged challenges, we extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference. Specifically, to alleviate input insufficiency, we introduce ScenePick, a frame sampling strategy that balances spatial coverage with object semantics to produce compact yet comprehensive scene representations. In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations, thereby aligning optimization with spatial accuracy. Extensive experiments across multiple benchmarks demonstrate consistent improvements over strong baselines, while ablation studies validate both the individual and joint contributions of each component, and further analyses provide guidance for future research.

Problem

• 3D spatial reasoning
• vision-language models
• spatial uncertainty
• video understanding
• spatial constraints

Innovation

• 3D spatial reasoning
• frame sampling
• spatial constraints
• vision-language models
• video understanding