Warehouse Spatial Question Answering with LLM Agent

2025-07-14

AI Summary

Current multimodal large language models (MLLMs) exhibit severe limitations in spatial understanding and reasoning within complex indoor warehouse environments, failing to accurately answer questions regarding object location, count, and inter-object distances. To address this, we propose a data-efficient spatial question-answering agent framework that avoids large-scale fine-tuning. Our method integrates multimodal input parsing, an explicit spatial reasoning module, coordinated tool invocation (e.g., 3D localization APIs), and an LLM-based agent architecture to decompose, reason about, and execute spatial queries. The core innovation lies in injecting symbolic spatial knowledge—such as geometric constraints and topological relations—directly into the LLM’s decision-making pipeline, thereby enhancing its geometric perception and logical deduction capabilities. Evaluated on the 2025 AI City Challenge Warehouse dataset, our approach achieves state-of-the-art accuracy and real-time performance across object localization, counting, and distance estimation tasks.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatial understanding has been a challenging task for existing Multi-modal Large Language Models~(MLLMs). Previous methods leverage large-scale MLLM finetuning to enhance MLLM's spatial understanding ability. In this paper, we present a data-efficient approach. We propose a LLM agent system with strong and advanced spatial reasoning ability, which can be used to solve the challenging spatial question answering task in complex indoor warehouse scenarios. Our system integrates multiple tools that allow the LLM agent to conduct spatial reasoning and API tools interaction to answer the given complicated spatial question. Extensive evaluations on the 2025 AI City Challenge Physical AI Spatial Intelligence Warehouse dataset demonstrate that our system achieves high accuracy and efficiency in tasks such as object retrieval, counting, and distance estimation. The code is available at: https://github.com/hsiangwei0903/SpatialAgent

Problem

• Enhancing spatial understanding in MLLMs for warehouses
• Solving spatial QA in complex indoor environments
• Improving accuracy in object retrieval and distance estimation

Innovation

• LLM agent system for spatial reasoning
• Integrates multiple tools for interaction
• Achieves high accuracy in warehouse tasks