S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

2026-06-18

AI Summary

This work addresses the limited capacity of existing vision-language models (VLMs) and tool-augmented agents to perform effective spatial reasoning in continuous, dynamic 3D environments, as they are largely confined to static, single-frame understanding. We propose S-Agent, a novel paradigm that reframes the VLM as a semantic planner driven by hierarchical tool invocation. By orchestrating a spatial toolchain that integrates 2D perception, 3D geometric reconstruction, and spatiotemporal memory, S-Agent enables cross-frame evidence accumulation and high-level spatial knowledge construction without requiring any additional training. The framework also generates high-quality spatial reasoning trajectories suitable for supervised fine-tuning. Experiments demonstrate that S-Agent substantially improves both open- and closed-source VLMs on multiview and video-based spatial reasoning benchmarks. Fine-tuning on the S-300K dataset generated by our method yields the S-Agent-8B model, which outperforms same-scale baselines and rivals advanced closed-source systems such as GPT-5.4 and Gemini 3.

Abstract

Real-world spatial intelligence requires reasoning over a continuous and evolving 3D world, yet existing VLMs and tool-augmented agents largely remain tied to static, stateless inference from isolated visual observations. We introduce \textbf{\textsc{S-Agent}}, a spatial tool-use agentic paradigm for understanding and reasoning over continuous multi-view images and videos. By formulating spatial reasoning as spatio-temporal evidence accumulation rather than isolated frame-level prediction, \textsc{S-Agent} reshapes spatial perception into scene-centric understanding beyond frame-centric recognition. Specifically, \textsc{S-Agent} casts the VLM as a semantic planner that decides what evidence is needed, while a hierarchy of spatial tools and experts grounds objects in 2D, lifts them into 3D geometric evidence, and aggregates this evidence into high-level spatial knowledge (\textit{e.g.}, counting, measurement, orientation, and relative position). Additionally, a temporal memory mechanism, including Scene Memory for maintaining the evolving scene state and Agent Memory for accumulating reasoning context, enables evidence integration across frames and reasoning steps. Comprehensive experiments on multi-view and video spatial reasoning benchmarks show that \textsc{S-Agent} consistently improves both open-source and closed-source VLMs in a training-free manner. Beyond inference-time augmentation, supervised fine-tuning (SFT) on \textsc{S-Agent}-generated spatial trajectories \textsc{S-300K} yields \textsc{S-Agent-8B}, a compact spatial agent that significantly surpasses similar-scale baselines (e.g., Qwen3-VL-8B) and performs comparably to advanced closed-source models (e.g., GPT-5.4 and Gemini 3).

Problem

• spatial intelligence
• visual language models
• continuous 3D reasoning
• multi-view perception
• temporal reasoning

Innovation

• spatial reasoning
• tool-augmented agent
• 3D scene understanding
• temporal memory
• evidence accumulation