Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models

2026-03-25

AI Summary

This work addresses the challenge of integrating 3D scene information into large language models while balancing expressive spatial relationship modeling with scalability. Existing approaches either rely on absolute positional encodings that lack geometric awareness or explicitly model pairwise object relationships, leading to quadratic growth in input length with respect to the number of objects. To overcome these limitations, we propose QuatRoPE, a quaternion-inspired rotational positional encoding that explicitly computes inter-object spatial relations through dot products within the attention mechanism. Furthermore, we introduce Isolated Gated RoPE Expansion (IGRE), a mechanism that precisely modulates the influence of the new encoding on the model’s pre-existing capabilities. Our method achieves geometrically consistent spatial reasoning with linear input complexity, enhancing 3D task performance without compromising linguistic competence, thereby offering both scalability and compatibility.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Spatial reasoning focuses on locating target objects based on spatial relations in 3D scenes, which plays a crucial role in developing intelligent embodied agents. Due to the limited availability of 3D scene-language paired data, it is challenging to train models with strong reasoning ability from scratch. Previous approaches have attempted to inject 3D scene representations into the input space of Large Language Models (LLMs) and leverage the pretrained comprehension and reasoning abilities for spatial reasoning. However, models encoding absolute positions struggle to extract spatial relations from prematurely fused features, while methods explicitly encoding all spatial relations (which is quadratic in the number of objects) as input tokens suffer from poor scalability. To address these limitations, we propose QuatRoPE, a novel positional embedding method with an input length that is linear to the number of objects, and explicitly calculates pairwise spatial relations through the dot product in attention layers. QuatRoPE's holistic vector encoding of 3D coordinates guarantees a high degree of spatial consistency, maintaining fidelity to the scene's geometric integrity. Additionally, we introduce the Isolated Gated RoPE Extension (IGRE), which effectively limits QuatRoPE's influence to object-related tokens, thereby minimizing interference with the LLM's existing positional embeddings and maintaining the LLM's original capabilities. Extensive experiments demonstrate the effectiveness of our approaches. The code and data are available at https://github.com/oceanflowlab/QuatRoPE.

Problem

• 3D spatial reasoning
• object relation encoding
• Large Language Models
• scalability
• positional embedding

Innovation

• QuatRoPE
• 3D spatial reasoning
• object relation encoding
• scalable positional embedding
• Large Language Models