OpenFusion++: An Open-vocabulary Real-time Scene Understanding System

2025-04-27

AI Summary

Existing real-time open-vocabulary 3D scene understanding methods suffer from low instance segmentation accuracy, static semantic updates, and weak responses to complex language queries. To address these challenges, this paper proposes the first real-time open-vocabulary 3D understanding system designed for dynamic environments. Our method introduces three key innovations: (1) an instance-area-adaptive semantic caching mechanism enabling global label evolution; (2) a dual-path cross-modal encoding framework that jointly models object attributes and environmental context; and (3) a robust perception pipeline integrating TSDF voxel reconstruction with foundation-model confidence map fusion. Extensive experiments on ICL, Replica, ScanNet, and ScanNet++ demonstrate significant improvements in semantic segmentation accuracy and open-ended query response latency, consistently outperforming state-of-the-art baselines across all major metrics.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Systems and Infrastructure for Web, Mobile and WoT: Location- and context-aware Web and WoT applications and services

Abstract

Real-time open-vocabulary scene understanding is essential for efficient 3D perception in applications such as vision-language navigation, embodied intelligence, and augmented reality. However, existing methods suffer from imprecise instance segmentation, static semantic updates, and limited handling of complex queries. To address these issues, we present OpenFusion++, a TSDF-based real-time 3D semantic-geometric reconstruction system. Our approach refines 3D point clouds by fusing confidence maps from foundational models, dynamically updates global semantic labels via an adaptive cache based on instance area, and employs a dual-path encoding framework that integrates object attributes with environmental context for precise query responses. Experiments on the ICL, Replica, ScanNet, and ScanNet++ datasets demonstrate that OpenFusion++ significantly outperforms the baseline in both semantic accuracy and query responsiveness.

Problem

• Imprecise 3D instance segmentation in real-time scene understanding
• Static semantic updates limiting dynamic environment adaptation
• Limited handling of complex open-vocabulary queries

Innovation

• Fuses confidence maps for refined 3D point clouds
• Dynamically updates semantics via adaptive cache
• Dual-path encoding integrates object-environment context