Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation

2025-03-27

AI Summary

To address the severe cross-domain performance degradation in open-vocabulary semantic segmentation, this paper proposes SemLA—a training-free test-time domain adaptation framework. SemLA constructs a LoRA adapter semantic library and leverages CLIP’s semantic space alignment to perform nearest-neighbor retrieval and weighted fusion of adapters for each input image, dynamically synthesizing an image-level personalized segmentation model. It establishes the first “zero-training, semantics-driven” domain adaptation paradigm for open-vocabulary segmentation, offering inherent interpretability (via adapter contribution tracing), strong data privacy preservation, and efficient scalability. Evaluated across a comprehensive benchmark spanning 10 datasets and 20 domains, SemLA significantly outperforms existing methods, setting a new state-of-the-art for domain-adaptive open-vocabulary semantic segmentation.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-vocabulary semantic segmentation models associate vision and text to label pixels from an undefined set of classes using textual queries, providing versatile performance on novel datasets. However, large shifts between training and test domains degrade their performance, requiring fine-tuning for effective real-world applications. We introduce Semantic Library Adaptation (SemLA), a novel framework for training-free, test-time domain adaptation. SemLA leverages a library of LoRA-based adapters indexed with CLIP embeddings, dynamically merging the most relevant adapters based on proximity to the target domain in the embedding space. This approach constructs an ad-hoc model tailored to each specific input without additional training. Our method scales efficiently, enhances explainability by tracking adapter contributions, and inherently protects data privacy, making it ideal for sensitive applications. Comprehensive experiments on a 20-domain benchmark built over 10 standard datasets demonstrate SemLA's superior adaptability and performance across diverse settings, establishing a new standard in domain adaptation for open-vocabulary semantic segmentation.

Problem

• Addresses performance degradation in open-vocabulary semantic segmentation across domains
• Proposes training-free domain adaptation using dynamic LoRA adapter fusion
• Enhances explainability and privacy for sensitive real-world applications

Innovation

• Training-free domain adaptation via Semantic Library
• Dynamic LoRA adapter fusion using CLIP embeddings
• Privacy-preserving scalable open-vocabulary segmentation