Feature Representation Transferring to Lightweight Models via Perception Coherence

2025-05-10

AI Summary

Lightweight student models suffer from limited representational capacity, making it difficult to replicate the geometric structure of teacher models. Method: This paper proposes a novel “perception-consistency” paradigm that preserves the teacher’s relative ranking of inter-sample differences—rather than enforcing strict feature-level alignment—and formally defines this concept. We introduce a ranking-aware loss function, provide theoretical analysis of representation transfer from a probabilistic perspective, and design mechanisms for modeling feature-space discrepancies and aligning distributions under weak constraints to ensure global discriminative consistency. Contribution/Results: Extensive experiments on multiple benchmarks demonstrate significant improvements in classification accuracy and generalization performance for lightweight architectures—including MobileNetV2 and ShuffleNet—consistently surpassing or matching state-of-the-art knowledge distillation methods.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

In this paper, we propose a method for transferring feature representation to lightweight student models from larger teacher models. We mathematically define a new notion called extit{perception coherence}. Based on this notion, we propose a loss function, which takes into account the dissimilarities between data points in feature space through their ranking. At a high level, by minimizing this loss function, the student model learns to mimic how the teacher model extit{perceives} inputs. More precisely, our method is motivated by the fact that the representational capacity of the student model is weaker than the teacher model. Hence, we aim to develop a new method allowing for a better relaxation. This means that, the student model does not need to preserve the absolute geometry of the teacher one, while preserving global coherence through dissimilarity ranking. Our theoretical insights provide a probabilistic perspective on the process of feature representation transfer. Our experiments results show that our method outperforms or achieves on-par performance compared to strong baseline methods for representation transferring.

Problem

• Transfer feature representation to lightweight student models
• Define perception coherence for better model relaxation
• Preserve global coherence through dissimilarity ranking

Innovation

• Transfer feature representation via perception coherence
• Use ranking-based loss for dissimilarity preservation
• Relax absolute geometry, maintain global coherence