Instruction Distillation: Text Instructions as Visual Examples

2026-08-27

AI Summary

为解决视觉上下文学习中大K设置成本高的问题，提出指令蒸馏方法，通过生成每张训练图像的结构化识别指令来减少令牌消耗和推理延迟。

Abstract

Visual in-context learning (ICL) with multimodal large language models (MLLMs) is effective for fine-grained visual classification, but each retrieved image example consumes several hundred context tokens, making large-$K$ settings prohibitively expensive at inference scale. We propose Instruction Distillation: an offline procedure in which the MLLM itself generates, for each individual training image, a structured identification instruction encoding general appearance cues, features that differentiate the class from visually similar ones, and a common confusion point. Unlike prior work that produces a single description per class, our instructions are generated per training image, preserving the intra-class visual diversity that per-class descriptions collapse. At inference time, we study five configurations sharing a single CLIP retrieval index: zero-shot, image ICL, instruction-only ICL, and two hybrid variants in which retrieved neighbors are split between images and instructions. Across seven fine-grained benchmarks and two MLLM backbones, instruction based pipelines match, or exceeds image ICL at $K{=}1$ and reduces per-query tokens by $2.9\times$ and inference latency by $3.3\times$ at $K{=}5$. Hybrid configurations further show that visual and textual ICL signals are complementary, images give visual patterns to learn and see, while instructions give explicit rules and logic. When both of these are provided, the quality of context improves, which is noticeable in the performance.

Problem

• Visual in-context learning
• multimodal large language models
• fine-grained visual classification
• context tokens
• inference scale

Innovation

• Instruction Distillation
• Multimodal Large Language Models (MLLMs)
• Fine-grained Visual Classification
• Intra-class Diversity
• Context Tokens