MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

2026-08-26

AI Summary

为解决CLIP在组合性识别上的不足，提出MLLMCLIP框架，通过特征级蒸馏直接从生成式多模态大语言模型向判别式CLIP传递知识。

Abstract

Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.

Problem

• vision-language models
• compositionality
• attribute-object
• relational structures

Innovation

• Feature-Level Distillation
• Heterogeneous Distillation Framework
• Attention-based Token Selection
• CKA-based Distillation Loss