Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding

2025-12-02

AI Summary

Existing 3D scene-language understanding methods employ standard causal attention masks, which impose artificial sequential ordering on inherently unordered 3D objects and restrict direct object–instruction interactions, thereby impairing task-oriented reasoning. To address this, we propose a dual masking mechanism: (1) a geometry-adaptive mask that dynamically models inter-object geometric relationships based on 3D spatial density distributions, eliminating sequence dependency; and (2) an instruction-aware mask that explicitly encodes language-guided attention over objects—without modifying model architecture or introducing additional parameters. This plug-and-play masking strategy significantly improves performance across multiple 3D language benchmarks, including ScanQA and ScanRefer. Our results demonstrate that attention mask design explicitly grounded in spatial structure is critical for effective multimodal reasoning in 3D vision-language tasks.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in 3D scene-language understanding have leveraged Large Language Models (LLMs) for 3D reasoning by transferring their general reasoning ability to 3D multi-modal contexts. However, existing methods typically adopt standard decoders from language modeling, which rely on a causal attention mask. This design introduces two fundamental conflicts in 3D scene understanding: sequential bias among order-agnostic 3D objects and restricted object-instruction attention, hindering task-specific reasoning. To overcome these limitations, we propose 3D Spatial Language Instruction Mask (3D-SLIM), an effective masking strategy that replaces the causal mask with an adaptive attention mask tailored to the spatial structure of 3D scenes. Our 3D-SLIM introduces two key components: a Geometry-adaptive Mask that constrains attention based on spatial density rather than token order, and an Instruction-aware Mask that enables object tokens to directly access instruction context. This design allows the model to process objects based on their spatial relationships while being guided by the user's task. 3D-SLIM is simple, requires no architectural modifications, and adds no extra parameters, yet it yields substantial performance improvements across diverse 3D scene-language tasks. Extensive experiments across multiple benchmarks and LLM baselines validate its effectiveness and underscore the critical role of decoder design in 3D multi-modal reasoning.

Problem

• Addresses sequential bias in 3D object processing
• Enables direct attention between objects and instructions
• Improves 3D scene-language reasoning without architectural changes

Innovation

• Replaces causal mask with adaptive spatial attention mask
• Geometry-adaptive mask uses spatial density not token order
• Instruction-aware mask links object tokens to task context