Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation

2025-01-22

AI Summary

In compositional image retrieval (CIR), the scarcity of high-quality training triplets—comprising a reference image, a textual modification, and a target image—has long been hindered by heavy reliance on costly manual annotation. To address this, this work pioneers the integration of counterfactual image generation into CIR data construction, proposing a fully automatic, annotation-free triplet synthesis framework. Our method leverages controllable visual editing and text–image alignment modeling to generate semantically coherent zero-shot triplets, which are then distilled to train CIR models. Evaluated on FashionIQ and CUHK-PEDES benchmarks, our approach achieves +6.2% and +5.8% absolute improvements in R@10 over supervised baselines. This paradigm eliminates manual annotation bottlenecks, enabling scalable, diverse, and unlabeled training set construction—thereby establishing a novel low-resource CIR framework grounded in synthetic data generation and distillation.

Technology Category

• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Composed Image Retrieval (CIR) provides an effective way to manage and access large-scale visual data. Construction of the CIR model utilizes triplets that consist of a reference image, modification text describing desired changes, and a target image that reflects these changes. For effectively training CIR models, extensive manual annotation to construct high-quality training datasets, which can be time-consuming and labor-intensive, is required. To deal with this problem, this paper proposes a novel triplet synthesis method by leveraging counterfactual image generation. By controlling visual feature modifications via counterfactual image generation, our approach automatically generates diverse training triplets without any manual intervention. This approach facilitates the creation of larger and more expressive datasets, leading to the improvement of CIR model's performance.

Problem

• Combination Image Retrieval
• Accuracy and Efficiency
• Reduced Manual Annotation

Innovation

• Counterfactual Image Generation
• Composite Image Retrieval
• Automated Dataset Creation