CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehension

2026-02-22

AI Summary

This work addresses the performance limitations of multimodal large language models (MLLMs) in retrieval tasks, which often stem from a mismatch between output formats and optimization objectives, as well as the common trade-off where enhancing retrieval capabilities compromises generative performance. To overcome these challenges, the authors propose CREM, a unified framework that jointly optimizes generative and contrastive objectives through a compression-driven representation enhancement mechanism. Key innovations include learnable chorus tokens, a compression-based prompt design, and a compression-aware attention mechanism. CREM achieves state-of-the-art performance on the MMEB retrieval benchmark while maintaining strong capabilities across diverse multimodal understanding and generation tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Multimodal Large Language Models (MLLMs) have shown remarkable success in comprehension tasks such as visual description and visual question answering. However, their direct application to embedding-based tasks like retrieval remains challenging due to the discrepancy between output formats and optimization objectives. Previous approaches often employ contrastive fine-tuning to adapt MLLMs for retrieval, but at the cost of losing their generative capabilities. We argue that both generative and embedding tasks fundamentally rely on shared cognitive mechanisms, specifically cross-modal representation alignment and contextual comprehension. To this end, we propose CREM (Compression-driven Representation Enhanced Model), with a unified framework that enhances multimodal representations for retrieval while preserving generative ability. Specifically, we introduce a compression-based prompt design with learnable chorus tokens to aggregate multimodal semantics and a compression-driven training strategy that integrates contrastive and generative objectives through compression-aware attention. Extensive experiments demonstrate that CREM achieves state-of-the-art retrieval performance on MMEB while maintaining strong generative performance on multiple comprehension benchmarks. Our findings highlight that generative supervision can further improve the representational quality of MLLMs under the proposed compression-driven paradigm.

Problem

• Multimodal Large Language Models
• retrieval
• generative capability
• representation alignment
• embedding-based tasks

Innovation

• compression-driven representation
• multimodal retrieval
• generative-contrastive alignment
• learnable chorus tokens
• multimodal large language models