RePair: Turning Retrieval Failures into Counterfactual Hard Pairs

2026-08-30

AI Summary

该论文针对视觉-语言检索中局部语义区分问题，提出RePair方法，通过修正模型错误生成的样本，形成难例正负对以优化训练。

Abstract

Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26\%--75\% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.

Problem

• vision-language retrieval
• CLIP-style dual encoders
• localized semantic distinctions
• hard-sample mining
• synthetic augmentation

Innovation

• counterfactual hard pairs
• LLM-guided editing
• local hard-pair contrastive objective