CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

2026-04-16

AI Summary

This work addresses the limitation of existing code retrieval models, which predominantly rely on plain text and fail to leverage the rich visual and structural information prevalent in programming contexts—such as UI mockups and UML diagrams—thereby hindering multimodal code understanding. To bridge this gap, we propose CodeMMR, the first unified multimodal code retrieval model capable of jointly processing natural language, source code, and images. CodeMMR employs an instruction-tuning-driven cross-modal alignment mechanism to embed all three modalities into a shared semantic space. We also introduce MMCoIR, the first large-scale multimodal code retrieval benchmark, spanning five visual domains, eight programming languages, and eleven libraries. Experimental results demonstrate that CodeMMR outperforms strong baselines—including UniIR, GME, and VLM2Vec—by an average of 10 percentage points in nDCG@10 and significantly enhances the fidelity and visual alignment of code generated by retrieval-augmented generation (RAG) systems on unseen tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Code search, framed as information retrieval (IR), underpins modern software engineering and increasingly powers retrieval-augmented generation (RAG), improving code discovery, reuse, and the reliability of LLM-based coding. Yet existing code IR models remain largely text-centric and often overlook the visual and structural aspects inherent in programming artifacts such as web interfaces, data visualizations, SVGs, schematic diagrams, and UML. To bridge this gap, we introduce MMCoIR, the first comprehensive benchmark for evaluating multimodal code IR across five visual domains, eight programming languages, eleven libraries, and show the challenge of the task through extensive evaluation. Therefore, we then propose CodeMMR, a unified retrieval model that jointly embeds natural language, code, and images into a shared semantic space through instruction-based multimodal alignment. CodeMMR achieves strong generalization across modalities and languages, outperforming competitive baselines (e.g., UniIR, GME, VLM2Vec) by an average of 10 points on nDCG@10. Moreover, integrating CodeMMR into RAG enhances code generation fidelity and visual grounding on unseen code generation tasks, underscoring the potential of multimodal retrieval as a core enabler for next-generation intelligent programming systems. Datasets are available at HuggingFace.

Problem

• code retrieval
• multimodal
• information retrieval
• visual programming artifacts
• unified retrieval

Innovation

• multimodal code retrieval
• instruction-based alignment
• shared semantic space
• retrieval-augmented generation
• code-image-language integration