Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network

2024-12-20

arXiv.org

AI Summary

Existing temporal sentence grounding (TSG) methods train on untrimmed video–sentence query pairs independently, neglecting inter-pair correlations—leading to knowledge redundancy, inefficient training, and limited generalization. This paper proposes a novel multi-pair joint TSG paradigm, enabling a single model to collaboratively optimize multiple video–query pairs simultaneously. To this end, we design a multi-threaded knowledge transfer network featuring: (i) cross-modal contrastive learning to strengthen fine-grained alignment; (ii) a dual-granularity prototype matching mechanism—operating at both object/phrase level (spatial) and action/sentence level (temporal); and (iii) adaptive threshold-based hard negative mining coupled with self-supervised representation learning. Extensive experiments on multiple benchmarks demonstrate substantial improvements in both grounding accuracy and inference efficiency, achieving new state-of-the-art performance. Ablation studies confirm the effectiveness of inter-pair knowledge transfer and the model’s strong generalization capability across diverse queries and videos.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Given some video-query pairs with untrimmed videos and sentence queries, temporal sentence grounding (TSG) aims to locate query-relevant segments in these videos. Although previous respectable TSG methods have achieved remarkable success, they train each video-query pair separately and ignore the relationship between different pairs. We observe that the similar video/query content not only helps the TSG model better understand and generalize the cross-modal representation but also assists the model in locating some complex video-query pairs. Previous methods follow a single-thread framework that cannot co-train different pairs and usually spends much time re-obtaining redundant knowledge, limiting their real-world applications. To this end, in this paper, we pose a brand-new setting: Multi-Pair TSG, which aims to co-train these pairs. In particular, we propose a novel video-query co-training approach, Multi-Thread Knowledge Transfer Network, to locate a variety of video-query pairs effectively and efficiently. Firstly, we mine the spatial and temporal semantics across different queries to cooperate with each other. To learn intra- and inter-modal representations simultaneously, we design a cross-modal contrast module to explore the semantic consistency by a self-supervised strategy. To fully align visual and textual representations between different pairs, we design a prototype alignment strategy to 1) match object prototypes and phrase prototypes for spatial alignment, and 2) align activity prototypes and sentence prototypes for temporal alignment. Finally, we develop an adaptive negative selection module to adaptively generate a threshold for cross-modal matching. Extensive experiments show the effectiveness and efficiency of our proposed method.

Problem

• Locate query-relevant segments in untrimmed videos using sentence queries
• Co-train video-query pairs to improve cross-modal representation understanding
• Align visual and textual representations spatially and temporally across pairs

Innovation

• Multi-Thread Knowledge Transfer Network for co-training
• Cross-modal contrast module for semantic consistency
• Prototype alignment strategy for spatial-temporal alignment