Spatial Reasoning via Modality Switching Between Language and Symbolic Representation

2026-06-30

AI Summary

This work addresses the performance limitations of large language models in multi-hop text-to-spatial reasoning tasks, which arise from their exclusive reliance on natural language. To overcome this, the authors propose a dynamic modality-switching mechanism that adaptively alternates between natural language and structured symbolic representations—such as grid-based formats—based on signals of model confidence and task complexity. This approach enables principled selection of multimodal reasoning pathways tailored to the demands of each subtask. Experimental results demonstrate that incorporating grid representations through this dynamic strategy can improve model performance by up to 42% compared to purely language-based reasoning, underscoring the critical role of adaptive modality selection in enhancing spatial reasoning capabilities.

Abstract

Human reasoning is inherently multimodal: when problems become difficult, we rarely think in words alone. We often externalize our reasoning by sketching diagrams or drawing grids to understand the underlying conceptual structure and avoid mistakes. Building on this premise, our research investigates: (a) whether grounding multi-hop textual-spatial stories into geometry-aware modalities, such as layouts or grids, improves reasoning compared to natural language-based inference; and (b) whether a model can decide when to rely on natural language reasoning and when to switch to a structured modality. We address these questions by introducing a switching metric based on trustworthiness and complexity signals, which estimates when grounding a spatial story into structure is likely to improve performance. This takes a first step toward principled modality selection in Large Language Model (LLM) reasoning. Across our settings, switching from natural language-based reasoning to a grid-based representation improves LLM performance by up to 42\%, highlighting the importance of modality choice in shaping reasoning outcomes.

Problem

• spatial reasoning
• modality switching
• symbolic representation
• large language models
• reasoning

Innovation

• modality switching
• spatial reasoning
• symbolic representation
• large language models
• structured reasoning