Privacy-Preserving Depth-Only Open-Vocabulary 3D Semantic Segmentation Via Uncertainty-Guided Test-Time Optimization

2026-07-01

AI Summary

This work proposes UTTO, a framework for open-vocabulary 3D semantic segmentation that operates solely on depth maps without requiring RGB images, thereby preserving privacy. UTTO is the first to leverage prediction uncertainty as a guiding signal during test-time optimization to dynamically correct unreliable semantic responses from foundation models. By effectively integrating geometric cues with open-vocabulary semantic priors, the method achieves strong performance without any additional training. Evaluated on ScanNet20/40/200 benchmarks, UTTO significantly outperforms existing single-modality approaches, demonstrating both the feasibility and superiority of depth-only open-vocabulary 3D segmentation.

Abstract

Privacy-preserving perception is a critical requirement for deploying 3D scene understanding systems in real-world indoor environments, yet it remains underexplored in open-vocabulary 3D semantic segmentation. Existing methods typically rely on obtaining rich semantic cues from RGB images, which may expose privacy-sensitive visual information. Depth-only 3D geometry provides a privacy-preserving alternative, but the absence of appearance-based semantic cues makes open-vocabulary predictions highly uncertain and less reliable. Under this setting, we propose to convert uncertainty into a guidance signal to identify unreliable semantic responses and use semantic priors from foundation models to regularize their refinement. We present UTTO, an uncertainty-guided test-time optimization framework for depth-only open-vocabulary 3D semantic segmentation. Without additional training, experiments on ScanNet20, ScanNet40, and ScanNet200 demonstrate that UTTO consistently improves depth-only open-vocabulary 3D segmentation and outperforms representative baselines under privacy-preserving conditions.

Problem

• Privacy-Preserving
• Depth-Only
• Open-Vocabulary
• 3D Semantic Segmentation
• Uncertainty

Innovation

• privacy-preserving
• depth-only
• open-vocabulary
• uncertainty-guided
• test-time optimization