Simple Semi-supervised Knowledge Distillation from Vision-Language Models via $mathbf{ exttt{D}}$ual-$mathbf{ exttt{H}}$ead $mathbf{ exttt{O}}$ptimization

2025-05-12

AI Summary

Efficiently transferring knowledge from large vision-language models (VLMs) to lightweight task-specific models under resource constraints remains challenging, especially with limited labeled data. Method: This paper proposes Dual-Head Optimization (DHO), a semi-supervised cross-modal knowledge distillation framework requiring only minimal annotations. Its core innovation is a decoupled dual-prediction-head architecture: one head is supervised by human-annotated labels, while the other is guided by VLM-generated soft labels—eliminating gradient conflict entirely. A linear fusion mechanism replaces multi-stage fine-tuning, drastically simplifying training. Results: On ImageNet, DHO achieves a +3.0% accuracy gain using only 1% labeled data, establishing new state-of-the-art performance. The method yields models with fewer parameters, higher training efficiency, strong generalization, and improved deployability.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Vision-language models (VLMs) have achieved remarkable success across diverse tasks by leveraging rich textual information with minimal labeled data. However, deploying such large models remains challenging, particularly in resource-constrained environments. Knowledge distillation (KD) offers a well-established solution to this problem; however, recent KD approaches from VLMs often involve multi-stage training or additional tuning, increasing computational overhead and optimization complexity. In this paper, we propose $mathbf{ exttt{D}}$ual-$mathbf{ exttt{H}}$ead $mathbf{ exttt{O}}$ptimization ($mathbf{ exttt{DHO}}$) -- a simple yet effective KD framework that transfers knowledge from VLMs to compact, task-specific models in semi-supervised settings. Specifically, we introduce dual prediction heads that independently learn from labeled data and teacher predictions, and propose to linearly combine their outputs during inference. We observe that $ exttt{DHO}$ mitigates gradient conflicts between supervised and distillation signals, enabling more effective feature learning than single-head KD baselines. As a result, extensive experiments show that $ exttt{DHO}$ consistently outperforms baselines across multiple domains and fine-grained datasets. Notably, on ImageNet, it achieves state-of-the-art performance, improving accuracy by 3% and 0.1% with 1% and 10% labeled data, respectively, while using fewer parameters.

Problem

• Deploying large vision-language models in resource-constrained environments
• Reducing computational overhead in knowledge distillation from VLMs
• Improving semi-supervised learning efficiency with dual-head optimization

Innovation

• Dual-head optimization for semi-supervised knowledge distillation
• Linear combination of labeled and teacher prediction heads
• Mitigates gradient conflicts in feature learning