Expand Heterogeneous Learning Systems with Selective Multi-Source Knowledge Fusion

2024-12-05

AI Summary

To address the challenge of constructing customized models under label scarcity and heterogeneous data and device environments, this paper proposes HaT—the first selective multi-source knowledge fusion framework. HaT relaxes the conventional knowledge distillation assumption that teacher models are fully reliable; instead, it jointly models teacher reliability and data heterogeneity to enable sample-level weighted knowledge fusion and quality-aware dynamic conditional injection. The method integrates knowledge distillation, multi-model selection, prediction confidence weighting, and knowledge quality assessment. Evaluated across multi-task, multimodal, and cross-device settings, HaT achieves up to a 16.5% improvement in model accuracy and reduces communication overhead by 39%. It significantly enhances rapid adaptation to new users or domains while improving system scalability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Expanding existing learning systems to provide high-quality customized models for more domains, such as new users, is challenged by the limited labeled data and the data and device heterogeneities. While knowledge distillation methods could overcome label scarcity and device heterogeneity, they assume the teachers are fully reliable and overlook the data heterogeneity, which prevents the direct adoption of existing models. To address this problem, this paper proposes a framework, HaT, to expand learning systems. It first selects multiple high-quality models from the system at a low cost and then fuses their knowledge by assigning sample-wise weights to their predictions. Later, the fused knowledge is selectively injected into the customized models based on the knowledge quality. Extensive experiments on different tasks, modalities, and settings show that HaT outperforms state-of-the-art baselines by up to 16.5% accuracy and saves up to 39% communication traffic.

Problem

• Expanding learning systems
• Overcoming data and device heterogeneities
• Selective multi-source knowledge fusion

Innovation

• Selective Multi-Source Knowledge Fusion
• Sample-wise Weight Assignment
• Customized Model Knowledge Injection