Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval

2025-04-11

AAAI Conference on Artificial Intelligence

AI Summary

To address semantic ambiguity in Partially Relevant Video Retrieval (PRVR) caused by inconsistent granularity between textual queries and video segments, this paper challenges the conventional one-to-one matching assumption and proposes an Ambiguity-constrained Representation Learning (ARL) framework. Methodologically, ARL introduces: (1) a dual-criterion ambiguous pair detection mechanism grounded in uncertainty and similarity; (2) joint optimization via multi-positive contrastive learning and dual triplet-margin loss; and (3) cross-model ambiguity-aware modeling coupled with fine-grained text-frame semantic alignment. Evaluated on multiple PRVR benchmarks, ARL consistently outperforms state-of-the-art methods. It effectively mitigates error propagation and semantic drift induced by ambiguous samples, thereby enhancing both retrieval robustness and accuracy.

Technology Category

• Computer Vision: Image and Video Retrieval
• Knowledge Representation and Reasoning: Argumentation
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Partially Relevant Video Retrieval~(PRVR) aims to retrieve a video where a specific segment is relevant to a given text query. Typical training processes of PRVR assume a one-to-one relationship where each text query is relevant to only one video. However, we point out the inherent ambiguity between text and video content based on their conceptual scope and propose a framework that incorporates this ambiguity into the model learning process. Specifically, we propose Ambiguity-Restrained representation Learning~(ARL) to address ambiguous text-video pairs. Initially, ARL detects ambiguous pairs based on two criteria: uncertainty and similarity. Uncertainty represents whether instances include commonly shared context across the dataset, while similarity indicates pair-wise semantic overlap. Then, with the detected ambiguous pairs, our ARL hierarchically learns the semantic relationship via multi-positive contrastive learning and dual triplet margin loss. Additionally, we delve into fine-grained relationships within the video instances. Unlike typical training at the text-video level, where pairwise information is provided, we address the inherent ambiguity within frames of the same untrimmed video, which often contains multiple contexts. This allows us to further enhance learning at the text-frame level. Lastly, we propose cross-model ambiguity detection to mitigate the error propagation that occurs when a single model is employed to detect ambiguous pairs for its training. With all components combined, our proposed method demonstrates its effectiveness in PRVR.

Problem

• Addresses ambiguity in text-video relevance for retrieval
• Detects ambiguous pairs via uncertainty and similarity criteria
• Enhances learning with multi-positive contrastive and triplet loss

Innovation

• ARL detects ambiguous pairs via uncertainty and similarity
• Hierarchical learning with multi-positive contrastive loss
• Cross-model ambiguity detection reduces error propagation