un$^2$CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIP

2025-05-30

AI Summary

CLIP exhibits limited capacity for fine-grained visual discrimination and dense prediction due to insufficient detail representation. To address this, we propose un²CLIP—a novel framework that, for the first time, injects prior knowledge from the generative model unCLIP into the CLIP image encoder via reverse decomposition, significantly enhancing visual detail capture without compromising cross-modal alignment. Our method comprises three key components: (i) conditional generation modeling grounded in unCLIP, (ii) reverse fine-tuning of the CLIP image encoder, and (iii) cross-modal embedding space consistency constraints. Extensive experiments demonstrate that un²CLIP consistently outperforms both the original CLIP and state-of-the-art variants across diverse benchmarks—including MMVP-VLM, open-vocabulary segmentation, and multimodal large language models—establishing a new paradigm for fine-grained vision-language understanding.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Contrastive Language-Image Pre-training (CLIP) has become a foundation model and has been applied to various vision and multimodal tasks. However, recent works indicate that CLIP falls short in distinguishing detailed differences in images and shows suboptimal performance on dense-prediction and vision-centric multimodal tasks. Therefore, this work focuses on improving existing CLIP models, aiming to capture as many visual details in images as possible. We find that a specific type of generative models, unCLIP, provides a suitable framework for achieving our goal. Specifically, unCLIP trains an image generator conditioned on the CLIP image embedding. In other words, it inverts the CLIP image encoder. Compared to discriminative models like CLIP, generative models are better at capturing image details because they are trained to learn the data distribution of images. Additionally, the conditional input space of unCLIP aligns with CLIP's original image-text embedding space. Therefore, we propose to invert unCLIP (dubbed un$^2$CLIP) to improve the CLIP model. In this way, the improved image encoder can gain unCLIP's visual detail capturing ability while preserving its alignment with the original text encoder simultaneously. We evaluate our improved CLIP across various tasks to which CLIP has been applied, including the challenging MMVP-VLM benchmark, the dense-prediction open-vocabulary segmentation task, and multimodal large language model tasks. Experiments show that un$^2$CLIP significantly improves the original CLIP and previous CLIP improvement methods. Code and models will be available at https://github.com/LiYinqi/un2CLIP.

Problem

• Enhances CLIP's ability to capture visual details
• Improves performance on dense-prediction tasks
• Maintains alignment with original text encoder

Innovation

• Inverts unCLIP to enhance CLIP details
• Aligns generative and discriminative model spaces
• Improves CLIP's dense-prediction and multimodal tasks