SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features

2026-08-11

AI Summary

This work addresses a fundamental limitation in combining quantization-aware training (QAT) with knowledge distillation (KD) in label-free scenarios: the mismatch in feature ranges between teacher and student models introduces an irreducible lower bound on distillation loss. To resolve this, the authors propose SQuaT, a framework that applies the student’s quantization parameters to the teacher’s features, thereby performing student-aware quantization of teacher features and eliminating the residual error caused by range misalignment. SQuaT is the first method to theoretically remove this distillation loss lower bound, enabling significant performance gains in extreme low-bit settings (e.g., 1–2 bits) without requiring labels. Extensive experiments demonstrate that SQuaT consistently outperforms strong baselines across diverse architectures and quantization configurations, particularly excelling at ultra-low bitwidths, while maintaining broad applicability due to its architecture-agnostic design.

Abstract

Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation. However, in practical scenarios, training labels are often unavailable due to privacy, copyright, or cost constraints. Knowledge Distillation (KD) is a common approach to address this challenge, but we observe that prior work combining QAT with KD suffers from a fundamental limitation: during distillation, the range mismatch between the teacher and the quantized student model induces an unattainable residual, resulting in an irreducible lower bound on the distillation loss. Motivated by this observation, we propose SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound by applying the student's quantization parameters to quantize the teacher's features during distillation. Through comprehensive experiments across diverse settings, we demonstrate that SQuaT consistently outperforms strong baselines, with particularly pronounced gains in extreme low-bit (e.g., 1- and 2-bit) settings. Furthermore, extensive evaluations across various model design choices show that our approach does not rely on specific architectural assumptions, making it broadly applicable across diverse architectures and quantization settings. The source code is available at https://github.com/lcdbsa522/SQuaT.

Problem

• Quantization-Aware Training
• Knowledge Distillation
• Label-Free Learning
• Range Mismatch
• Quantized Neural Networks

Innovation

• Quantization-Aware Training
• Knowledge Distillation
• Self-Supervised Learning
• Low-Bit Quantization
• Student-Aware Quantization