Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models

2025-06-03

AI Summary

Vision-language models like CLIP excel at cross-modal alignment but suffer from limited fine-grained visual perception, hindering downstream multimodal large language models (MLLMs). To address this, we propose a kernel-based unsupervised visual embedding alignment method that aligns the CLIP visual encoder with the high-detail-aware DINOv2 encoder, while keeping CLIP’s text encoder frozen—requiring no textual supervision or image-text pairs. Our approach introduces a radial basis function (RBF) kernel-driven alignment framework in embedding space, enabling efficient stochastic optimization. Experiments demonstrate substantial improvements on zero-shot object recognition, fine-grained spatial reasoning, and localization tasks. When integrated into MLLMs such as LLaVA, our aligned visual encoder consistently enhances downstream multimodal understanding performance across diverse benchmarks, validating its generalizability and effectiveness without compromising CLIP’s pretrained linguistic capabilities.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP's limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP's visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance.

Problem

• Align CLIP visual representation with DINOv2 for better perception
• Enhance fine-grained details in vision-language model embeddings
• Improve zero-shot recognition and spatial reasoning in MLLMs

Innovation

• Kernel-based alignment of CLIP and DINOv2
• Enhances fine-grained visual perception
• Maintains text embedding compatibility