Multi-aspect Knowledge Distillation with Large Language Model

2025-01-23

AI Summary

Conventional image classification relies on single-label supervision, limiting models’ capacity to capture multidimensional semantic attributes—such as pose, texture, and spatial relationships—essential for robust visual understanding. Method: This paper proposes a multimodal large language model (MLLM)-based multi-faceted knowledge distillation framework. It pioneers the use of MLLMs as structured knowledge sources, leveraging directed prompting to extract vision–language joint logits and expanding the model’s output space to support fine-grained semantic modeling. A hybrid loss—combining categorical cross-entropy and binary cross-entropy—is introduced to enable effective multi-faceted knowledge transfer. Contribution/Results: The method achieves consistent and significant improvements over strong baselines on both image classification and object detection benchmarks. It markedly enhances model generalization and robustness under distribution shifts and adversarial perturbations, empirically validating the efficacy of integrating multidimensional semantic knowledge into visual representation learning.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Recent advancements in deep learning have significantly improved performance on computer vision tasks. Previous image classification methods primarily modify model architectures or add features, and they optimize models using cross-entropy loss on class logits. Since they focus on classifying images with considering class labels, these methods may struggle to learn various emph{aspects} of classes (e.g., natural positions and shape changes). Rethinking the previous approach from a novel view, we propose a multi-aspect knowledge distillation method using Multimodal Large Language Models (MLLMs). Our approach involves: 1) querying Large Language Model with multi-aspect questions relevant to the knowledge we want to transfer to the model, 2) extracting corresponding logits from MLLM, and 3) expanding the model's output dimensions to distill these multi-aspect logits. We then apply cross-entropy loss to class logits and binary cross-entropy loss to multi-aspect logits. Through our method, the model can learn not only the knowledge about visual aspects but also the abstract and complex aspects that require a deeper understanding. We primarily apply our method to image classification, and to explore the potential for extending our model, we expand it to other tasks, such as object detection. In all experimental results, our method improves the performance of the baselines. Additionally, we analyze the effect of multi-aspect knowledge distillation. These results demonstrate that our method can transfer knowledge about various aspects to the model and the aspect knowledge can enhance model performance in computer vision tasks. This paper demonstrates the great potential of multi-aspect knowledge distillation, and we believe it offers a promising direction for future research in computer vision and beyond.

Problem

• Computer Vision
• Object Recognition
• Abstract Attributes Understanding

Innovation

• Multi-attribute Knowledge Distillation
• Large Language Models
• Computer Vision Enhancement