Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

2026-03-22

AI Summary

This work addresses the challenge of adaptively balancing data supervision and teacher guidance in knowledge distillation for multimodal large language models, particularly when facing noisy samples or uncertain teacher predictions. To this end, the authors propose Beta-weighted Knowledge Distillation (Beta-KD), which, from a Bayesian perspective, models teacher guidance as a Gibbs prior over student activations, enabling adaptive modulation of reliance on the teacher. Beta-KD introduces, for the first time, an uncertainty-aware adaptive weighting mechanism that supports arbitrary distillation objectives and their combinations, while also providing a closed-form solution for computational efficiency. Extensive experiments on multiple multimodal visual question answering (VQA) benchmarks demonstrate that Beta-KD significantly outperforms existing distillation approaches and effectively enhances student model performance.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher--student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods. The code is available at https://github.com/Jingchensun/beta-kd.

Problem

• knowledge distillation
• multimodal large language models
• teacher uncertainty
• data supervision
• uncertainty-aware

Innovation

• Uncertainty-Aware
• Knowledge Distillation
• Bayesian Perspective
• Adaptive Weighting
• Multimodal Large Language Models