CLDA: Collaborative Learning for Enhanced Unsupervised Domain Adaptation

2024-09-04

arXiv.org

AI Summary

This paper addresses the degradation of non-salient parameter generalization in teacher models under unsupervised domain adaptation (UDA), caused by source-to-target domain shift, which misguides student model training. To tackle this, we propose a lightweight teacher-student collaborative learning framework for UDA. Our core contribution is the first introduction of a bidirectional dynamic correction mechanism: the student model performs real-time calibration of the teacher’s non-salient parameters via parameter saliency analysis, and the updated teacher model reciprocally enhances student training. Additionally, we integrate domain-invariant feature alignment with bidirectional gradient interaction for joint optimization. Evaluated on GTA→Cityscapes and Synthia→Cityscapes benchmarks, our method improves teacher/student mIoU by 0.7%/1.4% and 0.8%/2.0%, respectively—outperforming state-of-the-art baselines significantly.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Unsupervised Domain Adaptation (UDA) endeavors to bridge the gap between a model trained on a labeled source domain and its deployment in an unlabeled target domain. However, current high-performance models demand significant resources, making deployment costs prohibitive and highlighting the need for compact, yet effective models. For UDA of lightweight models, Knowledge Distillation (KD) leveraging a Teacher-Student framework could be a common approach, but we found that domain shift in UDA leads to a significant increase in non-salient parameters in the teacher model, degrading model's generalization ability and transferring misleading information to the student model. Interestingly, we observed that this phenomenon occurs considerably less in the student model. Driven by this insight, we introduce Collaborative Learning for UDA (CLDA), a method that updates the teacher's non-salient parameters using the student model and at the same time utilizes the updated teacher model to improve UDA performance of the student model. Experiments show consistent performance improvements for both student and teacher models. For example, in semantic segmentation, CLDA achieves an improvement of +0.7% mIoU for the teacher model and +1.4% mIoU for the student model compared to the baseline model in the GTA-to-Cityscapes datasets. In the Synthia-to-Cityscapes dataset, it achieves an improvement of +0.8% mIoU and +2.0% mIoU for the teacher and student models, respectively.

Problem

• Addresses domain shift in Unsupervised Domain Adaptation (UDA)
• Reduces non-salient parameters in teacher models for better generalization
• Improves performance of lightweight models via collaborative learning

Innovation

• Collaborative Learning updates teacher with student
• Teacher-Student framework improves UDA performance
• Reduces non-salient parameters in teacher model