Lost in Translation? Vocabulary Alignment for Source-Free Domain Adaptation in Open-Vocabulary Semantic Segmentation

2025-09-18

AI Summary

This paper addresses the source-free domain adaptation (SFDA) challenge in open-vocabulary semantic segmentation, where source-domain data are unavailable. We propose VocAlign, a novel framework that tackles this problem through two key innovations: (1) a vocabulary alignment strategy that explicitly models semantic correspondences between source-category names and target-domain visual features; and (2) a student–teacher framework integrating Top-K class filtering and LoRA-based fine-tuning to enhance pseudo-label quality while balancing knowledge transfer capability and computational efficiency. Evaluated on Cityscapes, VocAlign achieves a +6.11 mIoU improvement over strong baselines. On zero-shot segmentation benchmarks, it significantly outperforms existing source-free methods, establishing the first efficient and robust SFDA solution for open-vocabulary scenarios. This work sets a new state-of-the-art benchmark for source-free open-vocabulary semantic segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We introduce VocAlign, a novel source-free domain adaptation framework specifically designed for VLMs in open-vocabulary semantic segmentation. Our method adopts a student-teacher paradigm enhanced with a vocabulary alignment strategy, which improves pseudo-label generation by incorporating additional class concepts. To ensure efficiency, we use Low-Rank Adaptation (LoRA) to fine-tune the model, preserving its original capabilities while minimizing computational overhead. In addition, we propose a Top-K class selection mechanism for the student model, which significantly reduces memory requirements while further improving adaptation performance. Our approach achieves a notable 6.11 mIoU improvement on the CityScapes dataset and demonstrates superior performance on zero-shot segmentation benchmarks, setting a new standard for source-free adaptation in the open-vocabulary setting.

Problem

• Addresses vocabulary misalignment in source-free domain adaptation
• Enhances pseudo-label generation with additional class concepts
• Reduces computational overhead while improving segmentation accuracy

Innovation

• Vocabulary alignment strategy for pseudo-label generation
• Low-Rank Adaptation for efficient fine-tuning
• Top-K class selection to reduce memory requirements