Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding

2026-06-18

AI Summary

This work addresses the challenge of unifying 2D semantics and 3D geometry in visual-language modeling for 3D scene understanding, a task where existing approaches often rely on explicit 3D inputs or additional encoders, leading to architectural fragmentation. The authors propose a novel framework that operates solely on posed RGB images and a single 2D vision encoder, leveraging reconstructed 3D scene occupancy as a geometric prior to align foreground 2D features into 3D space, followed by unified reasoning via a large language model. This approach is the first to enable occupancy-guided visual-language modeling using only 2D inputs, seamlessly integrating pretrained 2D semantic knowledge with 3D geometric structure. Experiments demonstrate state-of-the-art performance in multi-view occupancy prediction and competitive results on 3D visual question answering and dense captioning tasks, matching or surpassing methods that require explicit 3D input.

Abstract

Recently, vision-language models (VLMs) have made significant progress in 3D scene understanding, driving advances in applications such as embodied intelligence and robotic vision. However, existing approaches typically either rely directly on explicit 3D inputs (e.g., point clouds or RGB-D sequences), or introduce an additional 3D geometry encoder to derive 3D-aware visual tokens from 2D images. Such designs structurally decouple 3D geometric perception from the rich 2D semantics learned via vision-language pre-training, hindering the development of a unified 3D vision-language representation. In this work, we propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder. Specifically, Occ-VLM reconstructs 3D scene occupancy as an auxiliary geometric prior, which is utilized to spatially associate foreground 2D tokens with 3D space. These tokens are then decoded by a Large Language Model (LLM) for unified scene understanding. Extensive experiments demonstrate that Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning: it attains state-of-the-art performance on multi-view occupancy prediction, while performing on par with 3D-input VLMs on 3D Visual Question Answering (VQA) and 3D dense captioning benchmarks.

Problem

• 3D scene understanding
• vision-language models
• geometric perception
• semantic representation
• occupancy prediction

Innovation

• Occupancy Prediction
• Vision-Language Model
• 3D Scene Understanding
• 2D-to-3D Alignment
• Geometric Prior