From Templates to Natural Language: Generalization Challenges in Instruction-Tuned LLMs for Spatial Reasoning

2025-05-20

AI Summary

This work identifies a critical bottleneck in the generalization capability of instruction-tuned large language models (LLMs) for spatial grounding tasks: a significant performance drop—averaging 32% lower accuracy—when transferring from synthetic template-based instructions to real human-authored ones. Using object layout understanding and generation on a 2.5D grid as the evaluation scenario, the study is the first to systematically demonstrate the decisive impact of instruction format (synthetic vs. natural) on spatial reasoning generalization. We propose a fine-grained error attribution framework that precisely isolates the semantic diversity and structural flexibility underlying complex instruction degradation. Furthermore, we construct a multi-granularity instruction generalization benchmark and empirically validate the fundamental limitations of current instruction-tuning paradigms in modeling spatial semantics. Results reveal that instruction fidelity—not just task coverage—is essential for robust spatial reasoning, challenging prevailing assumptions about instruction tuning efficacy in geometric and relational reasoning domains.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Instruction-tuned large language models (LLMs) have shown strong performance on a variety of tasks; however, generalizing from synthetic to human-authored instructions in grounded environments remains a challenge for them. In this work, we study generalization challenges in spatial grounding tasks where models interpret and translate instructions for building object arrangements on a $2.5$D grid. We fine-tune LLMs using only synthetic instructions and evaluate their performance on a benchmark dataset containing both synthetic and human-written instructions. Our results reveal that while models generalize well on simple tasks, their performance degrades significantly on more complex tasks. We present a detailed error analysis of the gaps in instruction generalization.

Problem

• Generalizing from synthetic to human-authored spatial instructions
• Assessing LLM performance on complex spatial grounding tasks
• Analyzing gaps in instruction generalization for object arrangements

Innovation

• Fine-tune LLMs with synthetic instructions
• Evaluate on synthetic and human-written instructions
• Analyze gaps in spatial instruction generalization