SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts

2026-07-07

AI Summary

Existing multimodal large language models struggle to bridge the representational gap between 2D semantics and 3D geometry, often relying on costly 3D data or RGB-only inputs that lead to spatiotemporal discontinuities and modality conflicts. This work proposes SpaR3D-MoE, an end-to-end framework that leverages only sparse RGB views to construct a geometry-aware graph via adaptive spatiotemporal manifold sampling and introduces a heterogeneous Mixture-of-Experts architecture with instruction- and pose-aware routing for efficient 3D spatial reasoning. Without dense 3D supervision, the method substantially mitigates modality conflict and achieves state-of-the-art performance across VSI-Bench, ScanQA, and SQA3D benchmarks. Notably, it attains an average score of 63.5 on VSI-Bench—outperforming the strongest baseline by 7.8 points—and improves path planning and relative direction tasks by 35.4% and 51.4%, respectively.

Abstract

Recent Multimodal Large Language Models (MLLMs) struggle to bridge the representational gap between 2D semantic understanding and 3D spatial geometry. Existing 3D-aware models either rely on costly 3D-specific data or utilize RGB-only inputs with heuristic sampling and monolithic, shallow fusion, which respectively disrupt essential spatiotemporal connectivity and induce modality contention across diverse spatial tasks. To overcome these bottlenecks, we introduce SpaR3D-MoE, an end-to-end framework that enables adaptive spatial reasoning by equipping MLLMs with geometry-aware capabilities from only sparse RGB inputs. First, we propose an adaptive spatiotemporal manifold sampling mechanism that constructs a geometry-aware spatiotemporal graph to extract informative keyframes, effectively mitigating sequence redundancy while preserving the scene's topological connectivity. Second, we introduce the heterogeneous geometry-inductive Mixture-of-Experts driven by an instruction-pose aware router, which adaptively routes multimodal tokens to specialized experts, resolving the cross-modal contention inherent in monolithic fusion. Extensive experiments on VSI-Bench, ScanQA, and SQA3D demonstrate that our method achieves state-of-the-art performance. Notably, SpaR3D-MoE achieves the highest average score of 63.5 on VSI-Bench, outperforming the strongest baseline by 7.8 absolute points, alongside relative improvements of 35.4% and 51.4% in Route Plan and Relative Direction tasks, respectively.

Problem

• 3D spatial reasoning
• multimodal large language models
• spatiotemporal connectivity
• modality contention
• sparse views

Innovation

• Mixture-of-Experts
• 3D spatial reasoning
• sparse views
• geometry-aware modeling
• adaptive routing