Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning

2024-05-28

AI Summary

This work addresses the challenge of enhancing model generalization in source-free domain adaptation (SFDA), where only unlabeled target-domain data is available. We propose Reliability-driven Curriculum Learning (RCL), a novel framework that introduces the first multimodal large language model (MLLM)-based reliability assessment mechanism for SFDA. RCL establishes a three-stage progressive curriculum: (1) high-quality pseudo-label generation via collaborative MLLMs; (2) self-correcting knowledge expansion to mitigate instruction failure; and (3) multi-hot masking to refine label confidence, thereby overcoming MLLM-specific evaluation blind spots and generalization bottlenecks in SFDA. Evaluated on standard benchmarks including DomainNet, RCL achieves state-of-the-art performance—outperforming the best prior method by 9.4%—and significantly improves model adaptability and robustness under source-free conditions.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Source-Free Domain Adaptation (SFDA) aims to adapt a pre-trained source model to a target domain using only unlabeled target data. Current SFDA methods face challenges in effectively leveraging pre-trained knowledge and exploiting target domain data. Multimodal Large Language Models (MLLMs) offer remarkable capabilities in understanding visual and textual information, but their applicability to SFDA poses challenges such as instruction-following failures, intensive computational demands, and difficulties in performance measurement prior to adaptation. To alleviate these issues, we propose $ extbf{Reliability-based Curriculum Learning (RCL)}$, a novel framework that integrates multiple MLLMs for knowledge exploitation via pseudo-labeling in SFDA. Our framework incorporates Reliable Knowledge Transfer, Self-correcting and MLLM-guided Knowledge Expansion, and Multi-hot Masking Refinement to progressively exploit unlabeled data in the target domain. RCL achieves state-of-the-art (SOTA) performance on multiple SFDA benchmarks, e.g., $ extbf{+9.4%}$ on DomainNet, demonstrating its effectiveness in enhancing adaptability and robustness without requiring access to source data. Our code is available at: https://github.com/Dong-Jie-Chen/RCL.

Problem

• Adapt pre-trained models to target domains without labeled data.
• Overcome challenges in leveraging MLLMs for domain adaptation.
• Enhance adaptability and robustness in source-free domain adaptation.

Innovation

• Integrates MLLMs for pseudo-labeling in SFDA
• Uses Reliability-based Curriculum Learning framework
• Enhances adaptability without source data access