TextTeacher: What Can Language Teach About Images?

2026-05-21

AI Summary

This work addresses how to effectively leverage the semantic knowledge of language models to enhance visual model performance without increasing inference complexity. The authors propose TextTeacher, a lightweight approach that, for the first time, introduces text embeddings as auxiliary supervision during image classification training without requiring multimodal joint training. Specifically, semantic anchors are generated via a frozen pretrained text encoder combined with a lightweight projection module to guide visual representation learning, while the original model architecture remains unchanged at inference time. Using only image captions and a standard ViT backbone, TextTeacher achieves up to a 2.7 percentage point accuracy gain on ImageNet and an average transfer improvement of 1.0 percentage point—substantially outperforming visual knowledge distillation—or enables a 33% inference speedup at comparable accuracy.

Abstract

The platonic representation hypothesis suggests that sufficiently large models converge to a shared representation geometry, even across modalities. Motivated by this, we ask: Can the semantic knowledge of a language model efficiently improve a vision model? As an answer, we introduce TextTeacher, a simple auxiliary objective that injects text embeddings as additional information into image classification training. TextTeacher uses readily available image captions, a pre-trained and frozen text encoder, and a lightweight projection to produce semantic anchors that efficiently guide representations during training while leaving the inference-time model unchanged. On ImageNet with standard ViT backbones, TextTeacher improves accuracy by up to +2.7 percentage points (p.p.) and yields consistent transfer gains (on average +1.0 p.p.) under the same recipe and compute. It outperforms vision knowledge distillation, yielding more accuracy at a constant compute budget or similar accuracy, but 33% faster. Our analysis indicates that TextTeacher acts as a feature-space preconditioner, shaping deeper layers in the first stages of training, and aiding generalization by supplying complementary semantic cues. TextTeacher adds negligible overhead, requires no costly multimodal training of the target model and preserves the simplicity and latency of pure vision models. Project page with code and captions: https://nauen-it.de/publications/text-teacher

Problem

• language-vision alignment
• semantic knowledge transfer
• image classification
• multimodal representation
• model generalization

Innovation

• TextTeacher
• language-guided vision learning
• semantic anchors
• feature-space preconditioning
• efficient multimodal training