Grid-Preserving Knowledge Distillation: Transferring Convolutional Inductive Bias to Vision Transformers under Data Scarcity

2026-08-11

AI Summary

This work addresses the significant performance gap between Vision Transformers (ViTs) and convolutional neural networks (CNNs) in data-scarce scenarios, where existing knowledge distillation methods struggle to effectively transfer the inductive bias of CNNs. To this end, the authors propose iBKD, a novel framework that, for the first time, preserves and leverages the spatial grid structure of the teacher CNN throughout the entire ViT distillation process. By aligning grid-structured features between student and teacher via an inductive bias attention module—augmented with channel attention, deformable spatial attention, and a convolutional cross-attention mechanism used only during training—the method efficiently injects locality priors without introducing inference overhead. Evaluated across seven ViT backbones and six data-scarce benchmarks, iBKD consistently outperforms both locality-aware and general-purpose distillation approaches, with performance gains becoming more pronounced as data availability decreases.

Abstract

Vision Transformers underperform convolutional networks when training data is scarce, and distilling convolutional inductive biases from a CNN teacher is an effective remedy that leaves the deployed model unchanged. General-purpose feature distillation, however, transfers little in this setting. The pooling, flattening, and logit-space projections it inherits from CNN to CNN pipelines discard the spatial grid in which locality and translation equivariance are encoded, and unlike a convolutional student, a ViT cannot rebuild that structure on its own. In this paper, we propose iBKD, a distillation framework that preserves the grid along the entire transfer path. Its core module, the Inductive Bias Attention Module, aggregates every student layer onto the teacher grid with learned weights, sharpens structural cues with channel and deformable spatial attention, and injects them through convolutional cross-attention that operates between grids rather than between token sets. The module is used only during training, so the deployed model is an unmodified ViT with no inference overhead. Across seven Transformer backbones and six data-scarce benchmarks, iBKD outperforms both locality-guidance methods and general knowledge distillation baselines, and its margin widens as training data shrinks.

Problem

• Vision Transformers
• data scarcity
• knowledge distillation
• convolutional inductive bias
• spatial grid

Innovation

• knowledge distillation
• vision transformers
• inductive bias
• spatial grid preservation
• convolutional cross-attention