Problem
Research questions and friction points this paper is trying to address.
Cross-Architecture Knowledge Distillation
Vision Foundation Model
Lightweight Visual State Space Model
Tea Leaf Disease Classification
Edge Devices
Cross-Architecture Knowledge Distillation from a Vision Foundation Model to a Lightweight Visual State Space Model for Tea Leaf Disease Classification
🤖 AI Summary
研究通过跨架构知识蒸馏，将DINOv2模型的知识迁移到轻量级视觉状态空间模型中，以解决茶叶病害分类在边缘设备上的部署问题。
📝 Abstract
Automated tea leaf disease classification supports precision agriculture, yet deploying accurate models on edge devices remains challenging under tight compute budgets. Self-supervised vision foundation models such as DINOv2 provide strong features but are too large for field deployment, while lightweight models trained from scratch on small agricultural datasets often underfit. We study cross-architecture knowledge distillation (KD) from a fine-tuned DINOv2 teacher (Vision Transformer) to a compact bidirectional Visual State Space Model (LVSSM) student, an underexplored direction because the architectures use fundamentally different token-mixing mechanisms. We identify and fix two training-stability problems that prevent the from-scratch SSM student from learning on limited data: a single large patch-embedding convolution and a fusion layer that severs the residual path. With a progressive convolutional stem and gated bidirectional selective-scan block, the 4.45M-parameter student trains stably. Across three seeds, temperature-scaled logit distillation raises test accuracy from 92.32+/-2.14% to 95.41+/-1.17% (best single run: 96.20%; macro-F1: 94.45%), a +3.09 percentage-point mean gain. The student uses 5.0 times fewer parameters than the 22M-parameter teacher while retaining 98.3% of its accuracy. Ablations show that intermediate feature-alignment losses reduce accuracy, making simple logit-level KD the strongest configuration. A fair from-scratch comparison shows the gain is specific to students that start below the teacher. We report per-class metrics, confusion matrices, bootstrap confidence intervals, and FLOPs/latency measurements, and discuss limitations including the single-dataset scope and simplified non-official SSM implementation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Cross-Architecture Knowledge Distillation
Visual State Space Model
DINOv2
Progressive Convolutional Stem
Gated Bidirectional Selective-Scan Block
🔎 Similar Papers
No similar papers found.
