Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrieval

2025-09-17

AI Summary

Text-to-Image Person Retrieval (TIPR) suffers from insufficient cross-modal alignment: existing methods over-rely on hard negative mining while neglecting unmatched positive samples and lack explicit modeling and verification of local feature alignment. To address this, we propose the Full-Mode Fine-grained Alignment (FMFA) framework, which achieves synergistic optimization through three innovations: (1) Adaptive Similarity Distribution Matching (A-SDM), which calibrates global semantic distributions; (2) an Explicit Fine-grained Alignment (EFA) module that integrates sparsified similarity matrices with hard-coded local constraints to strengthen part-level correspondences; and (3) an implicit relational reasoning mechanism enabling end-to-end joint training. Evaluated on CUHK-PEDES, RSTPReid, and ICFG-PEDES benchmarks, FMFA significantly outperforms existing global-matching approaches, achieving state-of-the-art performance.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Multimodal Learning
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Text-to-Image Person Retrieval (TIPR) is a cross-modal matching task that aims to retrieve the most relevant person images based on a given text query. The key challenge in TIPR lies in achieving effective alignment between textual and visual modalities within a common latent space. To address this challenge, prior approaches incorporate attention mechanisms for implicit cross-modal local alignment. However, they lack the ability to verify whether all local features are correctly aligned. Moreover, existing methods primarily focus on hard negative samples during model updates, with the goal of refining distinctions between positive and negative pairs, often neglecting incorrectly matched positive pairs. To alleviate these issues, we propose FMFA, a cross-modal Full-Mode Fine-grained Alignment framework, which enhances global matching through explicit fine-grained alignment and existing implicit relational reasoning -- hence the term ``full-mode" -- without requiring additional supervision. Specifically, we design an Adaptive Similarity Distribution Matching (A-SDM) module to rectify unmatched positive sample pairs. A-SDM adaptively pulls the unmatched positive pairs closer in the joint embedding space, thereby achieving more precise global alignment. Additionally, we introduce an Explicit Fine-grained Alignment (EFA) module, which makes up for the lack of verification capability of implicit relational reasoning. EFA strengthens explicit cross-modal fine-grained interactions by sparsifying the similarity matrix and employs a hard coding method for local alignment. Our proposed method is evaluated on three public datasets, achieving state-of-the-art performance among all global matching methods. Our code is available at https://github.com/yinhao1102/FMFA.

Problem

• Achieving effective cross-modal alignment between text and images
• Verifying correct alignment of all local features in retrieval
• Addressing incorrectly matched positive pairs during model training

Innovation

• Explicit fine-grained alignment without extra supervision
• Adaptive similarity distribution matching for unmatched pairs
• Sparsified similarity matrix with hard coding