Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models

2026-04-16

AI Summary

Existing vision-language knowledge distillation methods struggle to achieve effective cross-modal knowledge alignment due to modality-independent supervision. This work proposes Switch-KD, a novel framework that enables vision-to-language knowledge transfer within a unified textual probability space. By introducing a pioneering visual switching mechanism, the student model’s visual outputs are mapped onto the teacher’s language pathway, while a dynamic bidirectional logits difference loss implicitly transfers visual knowledge and preserves distributional structure. Notably, Switch-KD requires no architectural modifications and efficiently distills knowledge from a 3B-parameter teacher model, yielding an average improvement of 3.6 points across ten benchmarks for TinyLLaVA, a compact 0.5B-parameter vision-language model.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.

Problem

• Vision-Language Models
• Knowledge Distillation
• Multimodal Alignment
• Modality-specific Supervision
• Cross-modal Knowledge Transfer

Innovation

• Knowledge Distillation
• Vision-Language Models
• Cross-modal Alignment
• Visual-Switch Distillation
• Dynamic Bi-directional Logits Difference