CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning

2026-04-03

AI Summary

This work addresses the limitations of existing vision-language models that rely on a single visual encoder, which struggles to simultaneously achieve effective cross-modal alignment and dense semantic understanding. To overcome this, the authors propose a modular fusion framework that systematically integrates contrastive (e.g., CLIP) and self-supervised (e.g., DINO) visual representations for the first time. The approach employs entropy-guided multi-layer aggregation and orthogonality-constrained projections to reduce redundancy, and introduces a RoPE-enhanced cross-attention mechanism to align heterogeneous token grids, yielding compact fused visual tokens suitable for integration with decoder-only large language models. The method substantially outperforms single-encoder baselines across multiple vision-language benchmarks, achieving an average gain of 4.9% on visual understanding tasks, a 5.4% improvement in referring expression comprehension, and state-of-the-art performance on the RefCOCO grounding task.

Abstract

Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.

Problem

• vision-language learning
• complementary encoders
• contrastive learning
• self-supervised learning
• visual representation fusion

Innovation

• multi-encoder fusion
• contrastive learning
• self-supervised vision
• representation-level fusion
• RoPE-enhanced cross-attention