Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge

2026-06-25

AI Summary

This work addresses the challenge of transferring high-level semantic knowledge from large language models (LLMs) to purely visual models for improved fine-grained classification performance and robustness, without relying on multimodal paired data. The authors propose LaViD, a novel framework that enables knowledge distillation from an LLM using only textual supervision: by prompting the LLM to generate multiple-choice questions that probe semantic distinctions among categories, they construct soft label distributions as “concept signatures.” These are integrated into training via an auxiliary distillation loss and logit normalization. Experiments demonstrate that LaViD outperforms existing methods such as MaKD, DKD, and MLKD across multiple fine-grained benchmarks and significantly improves worst-group accuracy on the Waterbirds dataset, effectively mitigating spurious correlations.

Abstract

Large Language Models (LLMs) possess broad conceptual knowledge acquired through large-scale text pretraining, yet their potential to supervise models in other modalities remains underexplored. In this work, we propose LaViD--Language-to-Visual Knowledge Distillation--a simple and effective framework for transferring high-level semantic knowledge from a language-only teacher to a vision-only student model. Instead of relying on paired multimodal data, LaViD elicits conceptual signals from an LLM by prompting it to generate multiple-choice questions (MCQs) that probe semantic distinctions between visual classes. Each class is mapped to a soft label distribution over these MCQs, forming a rich conceptual signature that guides the student through an auxiliary distillation loss. Notably, despite using a language-only teacher without access to image data, LaViD consistently outperforms recent methods like MaKD that distill from vision-language models across multiple fine-grained benchmarks. It also achieves competitive or superior performance compared to state-of-the-art visual distillation methods such as DKD and MLKD, with further gains when combined with logit standardization. On the Waterbirds dataset, LaViD substantially improves worst-group accuracy, demonstrating enhanced robustness to spurious correlations with distillation. Code is available at https://github.com/lliangthomas/lavid.

Problem

• cross-modality transfer
• knowledge distillation
• large language models
• fine-grained visual recognition
• language-to-visual

Innovation

• cross-modality knowledge distillation
• large language models
• fine-grained visual recognition
• semantic concept transfer
• robustness to spurious correlations