SENSE: Stereo OpEN Vocabulary SEmantic Segmentation

2026-04-17

AI Summary

This work addresses the limited spatial accuracy of existing open-vocabulary semantic segmentation methods, which predominantly rely on single-view images and struggle with occlusions and object boundaries. To overcome this, we propose the first open-vocabulary stereo semantic segmentation framework that integrates geometric cues from binocular images with semantic priors from vision-language models, enabling more precise natural language–driven scene understanding. Trained on our newly introduced PhraseStereo dataset, the proposed method achieves significant performance gains across multiple benchmarks: it improves average precision by 2.9% on PhraseStereo—surpassing the strongest baseline by 0.76%—and yields relative mIoU improvements of 3.5% and 18% on Cityscapes and KITTI, respectively.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-vocabulary semantic segmentation enables models to segment objects or image regions beyond fixed class sets, offering flexibility in dynamic environments. However, existing methods often rely on single-view images and struggle with spatial precision, especially under occlusions and near object boundaries. We propose SENSE, the first work on Stereo OpEN Vocabulary SEmantic Segmentation, which leverages stereo vision and vision-language models to enhance open-vocabulary semantic segmentation. By incorporating stereo image pairs, we introduce geometric cues that improve spatial reasoning and segmentation accuracy. Trained on the PhraseStereo dataset, our approach achieves strong performance in phrase-grounded tasks and demonstrates generalization in zero-shot settings. On PhraseStereo, we show a +2.9% improvement in Average Precision over the baseline method and +0.76% over the best competing method. SENSE also provides a relative improvement of +3.5% mIoU on Cityscapes and +18% on KITTI compared to the baseline work. By jointly reasoning over semantics and geometry, SENSE supports accurate scene understanding from natural language, essential for autonomous robots and Intelligent Transportation Systems.

Problem

• open-vocabulary semantic segmentation
• spatial precision
• occlusions
• object boundaries
• stereo vision

Innovation

• Stereo Vision
• Open-vocabulary Segmentation
• Vision-Language Models
• Geometric Reasoning
• Zero-shot Generalization