Unified Semantic Transformer for 3D Scene Understanding

2025-12-16

AI Summary

Current 3D scene understanding models are predominantly task-specific, lacking a unified framework for joint semantic-geometric reasoning across multiple dimensions. To address this, we propose UNITE—the first unified semantic Transformer architecture designed for real-world 3D scenes, accepting only RGB images as input and performing end-to-end joint prediction of 3D semantic segmentation, instance embeddings, open-vocabulary features, functional attributes, and joint constraints. UNITE introduces a novel multi-view consistency self-supervised loss coupled with 2D knowledge distillation, integrating feed-forward Transformers, geometric consistency modeling, and open-vocabulary representation learning. Evaluated on multiple 3D semantic understanding benchmarks, UNITE achieves state-of-the-art performance—outperforming dedicated single-task models and, in several metrics, even surpassing advanced methods that rely on ground-truth 3D geometry.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Holistic 3D scene understanding involves capturing and parsing unstructured 3D environments. Due to the inherent complexity of the real world, existing models have predominantly been developed and limited to be task-specific. We introduce UNITE, a Unified Semantic Transformer for 3D scene understanding, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model. Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry. Our approach is capable of directly predicting multiple semantic attributes, including 3D scene segmentation, instance embeddings, open-vocabulary features, as well as affordance and articulations, solely from RGB images. The method is trained using a combination of 2D distillation, heavily relying on self-supervision and leverages novel multi-view losses designed to ensure 3D view consistency. We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models, in many cases, surpassing methods that operate on ground truth 3D geometry. See the project website at unite-page.github.io

Problem

• Unifies diverse 3D semantic tasks in one model
• Predicts multiple semantic attributes from RGB images only
• Achieves state-of-the-art performance across various tasks

Innovation

• Unified transformer for multiple 3D semantic tasks
• End-to-end prediction from RGB images only
• Training with 2D distillation and multi-view consistency losses