OV-COAST: Cost Aggregation with Optimal Transport for Open-Vocabulary Semantic Segmentation

2025-06-04

AI Summary

To address the weak cross-domain generalization in open-vocabulary semantic segmentation (OVSS), this paper introduces optimal transport (OT) theory into the cost aggregation framework for the first time, proposing a two-stage vision-language feature alignment mechanism. First, a cost volume matrix is constructed from CLIP-extracted visual and textual features, and Sinkhorn distance is employed to compute the optimal transport plan. Second, this alignment serves as a differentiable supervision signal for training CAT-Seg. The method requires no additional annotations and significantly improves zero-shot transfer performance. On the MESS benchmark, the ViT-B-based CAT-Seg achieves a +1.72% mIoU gain over its baseline and outperforms SAN-B by 4.9%, establishing a new state-of-the-art. The core contribution lies in formulating optimal transport as a differentiable alignment prior, enabling robust, text-guided pixel-level matching across domains in open-set semantic segmentation.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Security and Privacy: Data transparency and provenance

Abstract

Open-vocabulary semantic segmentation (OVSS) entails assigning semantic labels to each pixel in an image using textual descriptions, typically leveraging world models such as CLIP. To enhance out-of-domain generalization, we propose Cost Aggregation with Optimal Transport (OV-COAST) for open-vocabulary semantic segmentation. To align visual-language features within the framework of optimal transport theory, we employ cost volume to construct a cost matrix, which quantifies the distance between two distributions. Our approach adopts a two-stage optimization strategy: in the first stage, the optimal transport problem is solved using cost volume via Sinkhorn distance to obtain an alignment solution; in the second stage, this solution is used to guide the training of the CAT-Seg model. We evaluate state-of-the-art OVSS models on the MESS benchmark, where our approach notably improves the performance of the cost-aggregation model CAT-Seg with ViT-B backbone, achieving superior results, surpassing CAT-Seg by 1.72 % and SAN-B by 4.9 % mIoU. The code is available at https://github.com/adityagandhamal/OV-COAST/}{https://github.com/adityagandhamal/OV-COAST/ .

Problem

• Enhancing open-vocabulary semantic segmentation generalization
• Aligning visual-language features via optimal transport theory
• Improving cost-aggregation model performance in OVSS tasks

Innovation

• Uses cost volume for optimal transport alignment
• Two-stage optimization with Sinkhorn distance
• Improves CAT-Seg model performance significantly