LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

2026-04-08

AI Summary

This work addresses the tendency of multimodal large language models to hallucinate and their lack of structured spatial priors when interpreting complex geometric layouts. To mitigate these limitations, the authors propose LAST, a novel framework that systematically integrates tool invocation into spatial reasoning. LAST introduces an interactive sandbox, LAST-Box, which abstracts heterogeneous visual tools—such as segmentation and depth estimation—into atomic instructions and reusable spatial skills, generating high-level multimodal prompts directly consumable by the model. A three-stage progressive training strategy further enhances the model’s ability to interpret and adaptively invoke tool outputs. Experiments demonstrate that LAST-7B outperforms baseline models by approximately 20% on average across four benchmarks, surpassing several strong closed-source counterparts and achieving state-of-the-art performance in complex spatial reasoning tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatial reasoning is a cornerstone capability for intelligent systems to perceive and interact with the physical world. However, multimodal large language models (MLLMs) frequently suffer from hallucinations and imprecision when parsing complex geometric layouts. As data-driven scaling struggles to internalize structured geometric priors and spatial constraints, integrating mature, specialized vision models presents a compelling alternative. Despite its promise, applying this paradigm to spatial reasoning is hindered by two key challenges: The difficulty of invoking heterogeneous, parameter-rich tools, as well as the challenge of understanding and effectively leveraging their diverse low-level outputs (e.g., segmentation masks, depth maps) in high-level reasoning. To address these challenges, we propose LAST, a unified framework for tool-augmented spatial reasoning. LAST features an extensible interactive sandbox, termed LAST-Box, which abstracts heterogeneous tool invocations into atomic instructions and reusable spatial skills, returning multimodal hints (e.g., annotated images and textual descriptions) that can be directly consumed by LLMs. We further design a three-stage progressive training strategy that guides models from understanding tool outputs to proficient and adaptive tool invocation. Experiments on four datasets show that LAST-7B achieves around 20\% performance gains over its backbone and outperforms strong proprietary closed-source LLMs, substantially enhancing reasoning on complex spatial tasks.

Problem

• spatial reasoning
• multimodal large language models
• tool integration
• geometric layout understanding
• hallucination

Innovation

• tool-augmented reasoning
• spatial reasoning
• multimodal large language models
• interactive sandbox
• progressive training