Learning a Semantic Calibration Network for Open-Vocabulary Semantic Segmentation

2026-06-06

AI Summary

This work addresses the limited generalization to novel categories and semantic ambiguity in open-vocabulary semantic segmentation by proposing a Semantic Calibration Network (SCN). Built upon the CLIP pretrained model, SCN employs cross-attention to generate vision-aware pseudo-text embeddings and incorporates a category disambiguation module together with a dynamic logical fusion mechanism to model inter-class semantic relationships and refine the mask classification process. While preserving CLIP’s strong zero-shot generalization capability, SCN significantly enhances discriminative performance. It achieves state-of-the-art results across multiple mainstream benchmarks, substantially improving segmentation accuracy and robustness in open-vocabulary settings.

Abstract

Semantic image segmentation assigns a predefined category label to each pixel, has achieved significant progress lately. Open-Vocabulary Segmentation (OVS) extends the segmentation task from a fixed set to an open set, enabling the identification and segmentation of novel concepts based on arbitrary text inputs, such as category names or descriptions. In this paper, we propose a novel Semantic Calibration Network (SCN) for open-vocabulary semantic segmentation. Different from prior approaches that focus on feature aggregation or simple fine-tuning of pre-trained models, SCN refines the mask classification process by explicitly modeling the semantic correlations between classes, aiming to enhance the model's discriminative power while effectively preserving the generalization abilities of the pre-trained CLIP model. Specifically, SCN comprises two core components: Class Disambiguation (CD) and Logits Fusion (LF). First, a cross-attention mechanism is utilized to transform the text embeddings into visually aware pseudo-text embeddings, in order to derive an enhanced similarity score that complements the original mask-text similarity score. Subsequently, the Class Disambiguation module captures implicit inter-class dependencies through a residual architecture to effectively resolve semantic ambiguities. Finally, the Logits Fusion module dynamically integrates multifaceted semantic evidence to ensure that the model achieves a robust semantic consensus while maintaining CLIP's inherent generalization capability. Comprehensive experimental results on mainstream benchmarks demonstrate that the proposed method achieves significant performance improvements compared to state-of-the-art algorithms.

Problem

• Open-Vocabulary Semantic Segmentation
• Semantic Ambiguity
• Novel Concept Segmentation
• Text-to-Image Alignment
• Zero-shot Segmentation

Innovation

• Semantic Calibration Network
• Open-Vocabulary Segmentation
• Class Disambiguation
• Logits Fusion
• CLIP