3D Segmentation Using Viewpoint-Dependent Spatial Relationships

2026-05-15

AI Summary

Existing methods for 3D referring expression segmentation struggle to accurately model viewpoint-dependent spatial relations such as “left/right” and “front/back” due to their neglect of the observer’s perspective. This work addresses this limitation by systematically introducing a viewpoint-conditioning mechanism that explicitly encodes camera pose to represent the observer’s viewpoint. We construct a large-scale, viewpoint-aware 3D referring expression dataset comprising 220,000 samples—scalable to tens of millions—leveraging camera poses to automatically annotate egocentric spatial relationships. By integrating this viewpoint information into 3D multimodal foundation models, our approach significantly enhances their comprehension of viewpoint-dependent instructions, improving segmentation mIoU from 0.30 to 0.47 and revealing a critical gap in current models’ ability to handle such tasks.

Abstract

Recent advances in 3D datasets and multimodal models have greatly improved natural language 3D scene understanding. However, most 3D referring segmentation methods do not explicitly represent the observer viewpoint, making spatial relations such as"left,""right,""front,"and"behind"ambiguous and difficult to evaluate. We introduce a viewpoint-aware 3D referring segmentation dataset containing 220k benchmark samples, and scalable to tens of millions of viewpoint-conditioned samples through dense viewpoint sampling. In this dataset, target objects can only be identified through observer-centric spatial relations, making viewpoint-conditioned grounding necessary. We construct the benchmark by leveraging camera poses to automatically annotate observer-centric relations (left/right, front/behind) together with viewpoint-independent relations (above/under). Using this benchmark, we evaluate several existing 3D large multimodal models in a zero-shot setting and find that current models struggle with viewpoint-dependent spatial instructions. We further study how explicit viewpoint information can be incorporated into 3D large multimodal models. We introduce a viewpoint representation that encodes camera poses and conditions the model on the observation viewpoint, improving segmentation accuracy on viewpoint-dependent relations and increasing mIoU from 0.30 to 0.47 compared to a model without viewpoint conditioning. The dataset, code, and trained models will be made publicly available upon acceptance.

Problem

• 3D referring segmentation
• viewpoint-dependent spatial relations
• observer-centric relations
• spatial ambiguity

Innovation

• viewpoint-aware
• 3D referring segmentation
• observer-centric spatial relations
• camera pose conditioning
• multimodal 3D understanding