MELT: Improve Composed Image Retrieval via the Modification Frequentation-Rarity Balance Network

2026-03-31

AI Summary

Existing compositional image retrieval methods are prone to frequency bias, often overlooking rare semantic concepts, and their similarity scoring lacks robustness in the presence of hard negatives and noise. To address these limitations, this work proposes a novel architecture that enhances attention to infrequent modification semantics through a frequency–rarity balancing network. It further integrates a multimodal attention mechanism with a diffusion-model-based denoising strategy to effectively mitigate two key challenges: asymmetric semantic localization and unreliable similarity estimation. Evaluated on two standard benchmarks, the proposed method significantly outperforms current state-of-the-art approaches, demonstrating its superiority in multimodal fusion and retrieval performance.

Abstract

Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of ``modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to ``Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at https://github.com/luckylittlezhi/MELT.

Problem

• Composed Image Retrieval
• frequency bias
• rare sample neglect
• hard negative samples
• similarity estimation

Innovation

• Composed Image Retrieval
• frequency-rarity balance
• rare semantic localization
• diffusion-based denoising
• hard negative samples