Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

2026-04-24

AI Summary

Existing generative multimodal embedding approaches often introduce redundant steps and semantic ambiguity during chain-of-thought (CoT) reasoning, degrading retrieval performance. To address this, this work proposes RIME, a rewriting-driven multimodal embedding framework that jointly optimizes generation and embedding through a retrieval-friendly text rewriting mechanism. RIME integrates cross-modal alignment (CMA) with a discriminative embedding-guided Refine-RL reinforcement learning strategy to significantly enhance embedding quality. Evaluated on the MMEB-V2, MRMR, and UVRB benchmarks, RIME substantially outperforms current generative embedding models while effectively reducing the length of reasoning texts, thereby achieving efficient and accurate multimodal cross-modal retrieval.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking.

Problem

• Chain-of-Thought
• generative multimodal embeddings
• semantic ambiguity
• redundant reasoning
• multimodal retrieval

Innovation

• Rewrite-driven Multimodal Embedding
• Cross-Mode Alignment
• Refine Reinforcement Learning
• Generative Multimodal Embeddings
• Retrieval-friendly Rewrite