Learning to Compose: Revisiting Proxy Task Design for Zero-Shot Composed Image Retrieval

2026-06-30

AI Summary

Existing zero-shot compositional image retrieval methods rely on predefined composition mechanisms, which struggle to model diverse and fine-grained semantic modifications. This work proposes FoCo, a novel framework that explicitly formulates the composition function as a learnable two-stage process: first, text-anchored visual aggregation focuses on visual content relevant to the textual modification; second, context-conditioned semantic completion integrates scene information to synthesize the target semantics. The model is jointly optimized through two proxy tasks and cross-instance contrastive learning, achieving state-of-the-art performance across four benchmarks. This approach substantially enhances both generalization capability and the diversity of semantic expression in compositional image retrieval.

Abstract

Composed Image Retrieval (CIR) retrieves a target image from a reference image and a textual modification. While supervised CIR relies on costly triplets, Zero-Shot CIR (ZS-CIR) alleviates this reliance through proxy tasks trained on image-text pairs. However, existing proxy tasks primarily enhance visual and textual representations to accommodate a predefined composition mechanism such as pseudo-word injection into a frozen text encoder or linear feature arithmetic. As a result, the composition function itself remains unlearned, limiting the model's ability to express diverse and fine-grained semantic modifications. To address this, we propose FoCo, which models composition as two coordinated stages: focusing on modification-relevant visual content, and then completing the target semantics. We realize these through two proxy tasks: text-anchored visual aggregation to selectively gather visual content guided by localized textual semantics, and context-conditioned semantic completion to transform these aggregated visuals with the remaining scene context into a coherent composed representation. The tasks are trained jointly with a cross-instance contrastive objective, encouraging semantic diversity and discouraging shortcut composition strategies. Extensive experiments on four ZS-CIR benchmarks show FoCo's state-of-the-art performance and improved generalization.

Problem

• Composed Image Retrieval
• Zero-Shot Learning
• Proxy Task
• Composition Function
• Semantic Modification

Innovation

• Zero-Shot Composed Image Retrieval
• Proxy Task Design
• Semantic Composition
• Visual-Textual Alignment
• Contrastive Learning