Monocular Open Vocabulary Occupancy Prediction for Indoor Scenes

2026-02-26

AI Summary

This work addresses the challenges of open-vocabulary 3D occupancy prediction in indoor scenes—characterized by rich semantics, dense geometry, and complex layouts—by introducing a novel paradigm that relies solely on geometric supervision. The proposed method employs 3D language-embedded Gaussians as a unified intermediate representation, effectively coupling fine-grained geometric structure with language-aligned semantic information. To stabilize training, an opacity-aware Poisson voxel aggregation operator is introduced, complemented by a progressive temperature decay strategy to mitigate feature mixing. Evaluated under a monocular input setting, the model achieves 59.50 IoU and 21.05 mIoU on Occ-ScanNet, significantly outperforming existing approaches and setting new state-of-the-art results, particularly on the open-vocabulary mIoU metric.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-vocabulary 3D occupancy is vital for embodied agents, which need to understand complex indoor environments where semantic categories are abundant and evolve beyond fixed taxonomies. While recent work has explored open-vocabulary occupancy in outdoor driving scenarios, such methods transfer poorly indoors, where geometry is denser, layouts are more intricate, and semantics are far more fine-grained. To address these challenges, we adopt a geometry-only supervision paradigm that uses only binary occupancy labels (occupied vs free). Our framework builds upon 3D Language-Embedded Gaussians, which serve as a unified intermediate representation coupling fine-grained 3D geometry with a language-aligned semantic embedding. On the geometry side, we find that existing Gaussian-to-Occupancy operators fail to converge under such weak supervision, and we introduce an opacity-aware, Poisson-based approach that stabilizes volumetric aggregation. On the semantic side, direct alignment between rendered features and open-vocabulary segmentation features suffers from feature mixing; we therefore propose a Progressive Temperature Decay schedule that gradually sharpens opacities during splatting, strengthening Gaussian-language alignment. On Occ-ScanNet, our framework achieves 59.50 IoU and 21.05 mIoU in the open-vocabulary setting, surpassing all existing occupancy methods in IoU and outperforming prior open-vocabulary approaches by a large margin in mIoU. Code will be released at https://github.com/JuIvyy/LegoOcc.

Problem

• open-vocabulary
• 3D occupancy
• indoor scenes
• monocular
• semantic understanding

Innovation

• open-vocabulary occupancy
• 3D Language-Embedded Gaussians
• geometry-only supervision
• opacity-aware Poisson aggregation
• Progressive Temperature Decay