Distilling Vision Transformers for Distortion-Robust Representation Learning

2026-04-24

AI Summary

This work addresses the challenge of learning distortion-robust visual representations in the absence of clean image data. The authors propose a novel asymmetric knowledge distillation framework that leverages a pre-trained Vision Transformer as a teacher model processing clean images and a student model handling distorted inputs. Through a multi-level alignment mechanism—encompassing global embeddings, patch-level features, and attention maps—the student is guided to approximate the representation space of the teacher. This approach uniquely integrates asymmetric distillation with hierarchical feature alignment, enabling high-quality representation learning using only distorted images. Extensive experiments demonstrate that the method significantly outperforms existing techniques across multiple distortion types and datasets on image classification tasks, achieving superior performance under equivalent human supervision.

Technology Category

• Computer Vision: Representation Learning for Vision
• Machine Learning: Representation Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision.

Problem

• distortion-robust representation
• self-supervised learning
• vision transformers
• knowledge distillation
• visual representation learning

Innovation

• asymmetric knowledge distillation
• distortion-robust representation
• Vision Transformer
• multi-level distillation
• self-supervised learning