Gradient-Attention Guided Dual-Masking Synergetic Framework for Robust Text-based Person Retrieval

2025-09-10

AI Summary

In text-driven person retrieval, CLIP-based methods suffer from weak fine-grained feature learning and high sensitivity to textual noise, primarily due to the lack of person-centric training data and insufficient global contrastive learning. To address these issues, this work proposes a novel co-optimization paradigm for data and model. First, leveraging context learning in multimodal large language models (MLLMs), we design a denoising data curation pipeline and release WebPerson—a large-scale, person-centric dataset. Second, we introduce GA-DMS, a gradient-attention-guided dual masking framework that adaptively masks noisy tokens and jointly optimizes masked token prediction with image-text contrastive learning. Extensive experiments demonstrate state-of-the-art performance on benchmarks including CUHK-PEDES and RSTPReid, achieving significant gains in retrieval accuracy and robustness against textual noise.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Image and Video Retrieval
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Although Contrastive Language-Image Pre-training (CLIP) exhibits strong performance across diverse vision tasks, its application to person representation learning faces two critical challenges: (i) the scarcity of large-scale annotated vision-language data focused on person-centric images, and (ii) the inherent limitations of global contrastive learning, which struggles to maintain discriminative local features crucial for fine-grained matching while remaining vulnerable to noisy text tokens. This work advances CLIP for person representation learning through synergistic improvements in data curation and model architecture. First, we develop a noise-resistant data construction pipeline that leverages the in-context learning capabilities of MLLMs to automatically filter and caption web-sourced images. This yields WebPerson, a large-scale dataset of 5M high-quality person-centric image-text pairs. Second, we introduce the GA-DMS (Gradient-Attention Guided Dual-Masking Synergetic) framework, which improves cross-modal alignment by adaptively masking noisy textual tokens based on the gradient-attention similarity score. Additionally, we incorporate masked token prediction objectives that compel the model to predict informative text tokens, enhancing fine-grained semantic representation learning. Extensive experiments show that GA-DMS achieves state-of-the-art performance across multiple benchmarks.

Problem

• Addresses scarcity of large-scale annotated person-centric vision-language data
• Overcomes limitations of global contrastive learning for fine-grained matching
• Reduces vulnerability to noisy text tokens in cross-modal retrieval

Innovation

• Automated MLLM-based data curation pipeline
• Gradient-attention guided adaptive text masking
• Masked token prediction for fine-grained learning