Noise May Contain Transferable Knowledge: Understanding Semi-supervised Heterogeneous Domain Adaptation from an Empirical Perspective

2025-02-19

AI Summary

This paper addresses the fundamental challenge in semi-supervised heterogeneous domain adaptation (SHDA): knowledge transfer under feature-space and distributional divergence between source and target domains, where only the source domain is fully labeled, the target domain provides only minimal labels, and no instance-level correspondences exist. Through a large-scale empirical study across 330 SHDA tasks, we first reveal that transferable knowledge resides primarily in the *transferability* and *discriminability* of source data—not semantic consistency or structural fidelity—even random noise can serve as effective source data. Building on this insight, we propose the Knowledge Transfer Framework (KTF), a unified approach for knowledge migration. Evaluated on multi-source heterogeneous tasks, KTF achieves performance on par with real source domains and significantly improves target-domain accuracy. The code and datasets are publicly released.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Semi-supervised heterogeneous domain adaptation (SHDA) addresses learning across domains with distinct feature representations and distributions, where source samples are labeled while most target samples are unlabeled, with only a small fraction labeled. Moreover, there is no one-to-one correspondence between source and target samples. Although various SHDA methods have been developed to tackle this problem, the nature of the knowledge transferred across heterogeneous domains remains unclear. This paper delves into this question from an empirical perspective. We conduct extensive experiments on about 330 SHDA tasks, employing two supervised learning methods and seven representative SHDA methods. Surprisingly, our observations indicate that both the category and feature information of source samples do not significantly impact the performance of the target domain. Additionally, noise drawn from simple distributions, when used as source samples, may contain transferable knowledge. Based on this insight, we perform a series of experiments to uncover the underlying principles of transferable knowledge in SHDA. Specifically, we design a unified Knowledge Transfer Framework (KTF) for SHDA. Based on the KTF, we find that the transferable knowledge in SHDA primarily stems from the transferability and discriminability of the source domain. Consequently, ensuring those properties in source samples, regardless of their origin (e.g., image, text, noise), can enhance the effectiveness of knowledge transfer in SHDA tasks. The codes and datasets are available at https://github.com/yyyaoyuan/SHDA.

Problem

• Understanding transferable knowledge in SHDA
• Exploring noise as source for knowledge transfer
• Developing unified Knowledge Transfer Framework (KTF)

Innovation

• Unified Knowledge Transfer Framework (KTF)
• Noise as transferable knowledge source
• Ensuring source sample transferability and discriminability