What Holds Back Open-Vocabulary Segmentation?

2025-08-06

AI Summary

Open-vocabulary segmentation is hindered by standard supervised paradigms, limiting generalization to unseen categories; meanwhile, existing vision-language pretraining approaches suffer from opaque transfer mechanisms, resulting in prolonged performance stagnation. To address this, we propose the first interpretable oracle analysis framework, introducing a ground-truth-guided bottleneck disentanglement module that systematically diagnoses fundamental deficiencies across three axes: cross-modal alignment, text-guided segmentation, and visual feature adaptation. Through large-scale, multi-model ablation studies, we quantitatively identify the inherent performance ceiling and its primary causes for the first time. Our work establishes a reproducible evaluation benchmark and shifts the research paradigm from empirical tuning to mechanism-driven design—thereby enabling controllable, principled optimization of open-vocabulary segmentation.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Standard segmentation setups are unable to deliver models that can recognize concepts outside the training taxonomy. Open-vocabulary approaches promise to close this gap through language-image pretraining on billions of image-caption pairs. Unfortunately, we observe that the promise is not delivered due to several bottlenecks that have caused the performance to plateau for almost two years. This paper proposes novel oracle components that identify and decouple these bottlenecks by taking advantage of the groundtruth information. The presented validation experiments deliver important empirical findings that provide a deeper insight into the failures of open-vocabulary models and suggest prominent approaches to unlock the future research.

Problem

• Open-vocabulary segmentation fails to recognize unseen concepts
• Performance plateaus due to unresolved bottlenecks
• Groundtruth-based oracle components address these limitations

Innovation

• Language-image pretraining on image-caption pairs
• Oracle components identify and decouple bottlenecks
• Groundtruth information improves open-vocabulary segmentation