Coarse Semantic Injection for LLM-Conditioned Structured Indoor Prediction

2026-05-16

AI Summary

This work addresses the limitations of existing point cloud encoders, which struggle to effectively represent thin structures such as doors and windows after voxelization and sparse pooling, and often miss isolated furniture instances in cluttered scenes. To overcome these challenges, the authors propose a semantic enhancement approach that compresses point-wise semantics into four coarse-grained categories—furniture, walls, openings, and others—and embeds them into the original point attributes via a novel RGBB four-state color encoding scheme. A lightweight routing-based semantic shift module is further introduced to fuse geometric and semantic information within a shared sparse token pathway, while preserving the architecture of downstream language model decoders. Experiments demonstrate significant improvements in opening localization accuracy and instance-level furniture detection performance on the Structured3D, SpatialLM, and ARKitScenes datasets.

Abstract

Large language models (LLMs) have recently been used as structured decoders for indoor understanding from 3D point-token inputs. However, point cloud encoders often under-represent thin structural elements such as doors and windows after voxelization and sparse pooling, and may miss individual furniture instances in cluttered scenes. We propose an interface-preserving semantic augmentation for LLM-conditioned structured decoding. The key idea is to associate semantic evidence with the point-cloud representation, reduce it to a coarse four-group code (furniture, walls, openings, and others), and encode it as an RGBB point interface: red for furniture, green for walls, blue for openings, and black for others, where RGBB denotes four semantic color states represented in three RGB channels rather than an additional fourth channel. This semantic color code is appended to the original raw point attributes before tokenization, so geometry and semantics share the same sparse tokenization path while the downstream language model decoder and output serialization remain unchanged. We further introduce a lightweight routed semantic shift module, with an auxiliary head used only for training-time ratio/budget regularization and analysis, to strengthen semantic cues after sparse pooling. The overall pipeline can use RGB-derived semantic evidence. Under these controlled semantic-source settings, the reported metrics improve across Structured3D, the SpatialLM dataset, and ARKitScenes, especially for opening localization and per-instance furniture detection in cluttered scenes. Ablations clarify the roles of semantic source, color coding, token fusion, and shift injection, while also showing that color/entropy effects remain nontrivial.

Problem

• structured indoor prediction
• point cloud encoding
• thin structural elements
• furniture instance detection
• semantic under-representation

Innovation

• semantic injection
• LLM-conditioned decoding
• RGBB encoding
• structured indoor prediction
• sparse tokenization