Weak-to-Strong Knowledge Distillation Accelerates Visual Learning

2026-04-16

AI Summary

Training large-scale vision models is computationally expensive, and existing knowledge distillation methods primarily focus on model compression or accuracy improvement rather than accelerating the training of strong models. This work proposes a plug-and-play weak-to-strong knowledge distillation strategy that leverages a fixed-weight weak teacher model during early training stages and dynamically terminates distillation once the student surpasses the teacher’s performance, significantly reducing the number of epochs needed to reach target accuracy. Notably, this is the first approach to employ knowledge distillation explicitly for accelerating strong model training rather than compression. The method demonstrates broad applicability across image classification, object detection, and diffusion-based generation tasks, achieving up to 4.8× epoch acceleration on ImageNet and CIFAR, 1.7× speedup on COCO detection, and a 2.5× reduction in FID-convergent steps for CIFAR-10 diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large-scale visual learning is increasingly limited by training cost. Existing knowledge distillation methods transfer from a stronger teacher to a weaker student for compression or final-accuracy improvement. We instead investigate distillation to accelerate the training of strong students. We propose a generalizable plug-and-play recipe that freezes a weaker teacher, applies distillation only in early training, and turns it off once the student reaches and surpasses teacher-level performance. For ImageNet and CIFAR classification, this strategy reaches target thresholds much earlier, with up to 4.8 times speedup measured by epochs. We confirm that the method generalizes to other tasks and report 1.7 times epoch speedup for object detection on the COCO dataset, and 2.5 times earlier target-FID crossing for diffusion generation on the CIFAR-10 dataset, measured in steps. These findings validate our method as a universal speedup mechanism for visual learning.

Problem

• knowledge distillation
• training acceleration
• visual learning
• strong student
• training cost

Innovation

• weak-to-strong distillation
• training acceleration
• knowledge distillation
• visual learning
• plug-and-play