SOLAR: Self-supervised Joint Learning for Symmetric Multimodal Retrieval

2026-05-15

AI Summary

This work addresses the challenge in symmetric multimodal retrieval (MM2MM), where queries and contexts are interchangeable yet existing approaches rely on annotated asymmetric data. We propose the first fully unsupervised, two-stage self-supervised framework that leverages unlabeled web image–text pairs to align shared semantics across modalities while preserving modality-specific characteristics through intersection mask modeling. The framework further incorporates a hard negative mining strategy to enhance embedding learning. To rigorously evaluate performance, we construct a high-quality, human-verified benchmark. Experimental results demonstrate that our method outperforms the strongest supervised vision-language models by 7.08 points on this new benchmark, while using over 50 times fewer parameters and a fivefold reduction in embedding dimensionality, thereby achieving substantial gains in both efficiency and effectiveness.

Abstract

In this work, we address the critical yet underexplored challenge of symmetric multimodal-to-multimodal (MM2MM) retrieval, where queries and contexts are interchangeable. Existing universal multimodal retrieval works struggle with this task, as they are constrained by the labeled asymmetric datasets used. We produce SOLAR (Self-supervised jOint LeArning for symmetric multimodal Retrieval), a novel two-stage self-supervised framework that leverages readily available unlabeled web-scale image-text pairs. Based on the observation that both semantic alignment and discrepancies exist between two modalities, in the first stage, we learn the intersection mask of image-text pair, allowing us to align intersection while preserving semantic of difference. In the second stage, the learned mask is further utilized to construct positive and hardnegative samples via masking different parts of image/text, which enable us to conduct self-supervised multimodal embedding learning. Complementing this framework, we present a new benchmark featuring high-quality human-verified positive and hard-negative pairs to evaluate symmetric MM2MM retrieval under realistic conditions, as well as the corresponding pipeline. Extensive experiments against ten SOTA methods show SOLAR surpasses the strongest supervised VLM by 7.08 points on this benchmark, with over 50x fewer model parameters and a 5x smaller embedding dimension. Code and benchmark will be available soon.

Problem

• symmetric multimodal retrieval
• multimodal-to-multimodal retrieval
• self-supervised learning
• image-text alignment
• hard-negative sampling

Innovation

• symmetric multimodal retrieval
• self-supervised learning
• intersection mask
• hard-negative mining
• multimodal embedding