LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models

2026-05-11

AI Summary

This work addresses the high computational and memory costs of deploying large vision-language models, a challenge exacerbated by the limited efficacy of conventional knowledge distillation when the capacity gap between teacher and student models is substantial. To overcome this, the authors propose a bottom-up cascaded knowledge distillation framework that introduces intermediate-sized teacher models to enable progressive, multi-stage knowledge transfer, mirroring the step-by-step learning process in human education. This approach innovatively structures distillation into successive phases, effectively mitigating knowledge transfer degradation caused by excessive model capacity disparity, and is supported by theoretical analysis based on generalization error bounds. Experiments built upon the LLaVA architecture demonstrate state-of-the-art performance across seven visual question answering benchmarks, achieving significantly higher accuracy for compact student models while maintaining efficient inference.

Abstract

Large Vision-Language Models (VLMs) are successful in addressing a multitude of vision-language understanding tasks, such as Visual Question Answering (VQA), but their memory and compute requirements remain a concern for practical deployment. A promising class of techniques for mitigating this concern is Knowledge Distillation, where knowledge from a high-capacity Teacher network is transferred to a considerably smaller Student network. However, the capacity gap between the two networks is both a blessing and a curse: the smaller the Student network, the better its efficiency, and the larger the Teacher, the more knowledge it carries; yet, beyond a point, the larger capacity gap between the two leads to worse knowledge transfer. To counter this effect, we propose a bottom-up cascaded knowledge distillation (CKD) framework. Instead of treating knowledge transfer as an activity involving one high-capacity Teacher (or an ensemble of such), inspired by human formal education systems, we introduce one (potentially, more) additional Teacher(s) of intermediate capacity that gradually bring the Student network to the next level, where the next (higher-capacity) Teacher can take over. We provide a theoretical analysis in order to study the effect of cascaded distillation in the generalization performance of the Student. We apply the proposed framework on models build upon the LLaVA methodology and evaluate the derived models on seven standard, publicly available VQA benchmarks, demonstrating their SotA performance.

Problem

• Vision-Language Models
• Knowledge Distillation
• Model Compression
• Capacity Gap
• Visual Question Answering

Innovation

• Cascaded Knowledge Distillation
• Vision-Language Models
• Model Compression
• LLaVA
• Visual Question Answering