good4cir: Generating Detailed Synthetic Captions for Composed Image Retrieval

2025-03-22

AI Summary

Fine-grained textual annotations for Composed Image Retrieval (CIR) are scarce, ambiguous, and lack diversity. Method: We propose the first end-to-end vision-language-driven synthetic annotation generation framework, featuring a novel three-stage structured pipeline: (i) fine-grained object parsing, (ii) cross-image comparable description generation, and (iii) semantic-aligned instruction synthesis—effectively mitigating hallucination while ensuring object-level consistency and modification diversity. Leveraging advanced vision-language models, the framework enables high-quality, cross-domain CIR dataset construction. Contribution/Results: Experiments demonstrate that our generated annotations significantly improve retrieval accuracy across multiple CIR benchmarks and enhance model generalization. The open-sourced framework has accelerated progress in multimodal retrieval research.

Technology Category

• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Composed image retrieval (CIR) enables users to search images using a reference image combined with textual modifications. Recent advances in vision-language models have improved CIR, but dataset limitations remain a barrier. Existing datasets often rely on simplistic, ambiguous, or insufficient manual annotations, hindering fine-grained retrieval. We introduce good4cir, a structured pipeline leveraging vision-language models to generate high-quality synthetic annotations. Our method involves: (1) extracting fine-grained object descriptions from query images, (2) generating comparable descriptions for target images, and (3) synthesizing textual instructions capturing meaningful transformations between images. This reduces hallucination, enhances modification diversity, and ensures object-level consistency. Applying our method improves existing datasets and enables creating new datasets across diverse domains. Results demonstrate improved retrieval accuracy for CIR models trained on our pipeline-generated datasets. We release our dataset construction framework to support further research in CIR and multi-modal retrieval.

Problem

• Generating detailed synthetic captions for composed image retrieval
• Overcoming dataset limitations with simplistic manual annotations
• Improving retrieval accuracy with fine-grained object descriptions

Innovation

• Generates synthetic captions using vision-language models
• Extracts fine-grained object descriptions from images
• Synthesizes textual instructions for image transformations