Multi-proposal Collaboration and Multi-task Training for Weakly-supervised Video Moment Retrieval

2026-05-14

AI Summary

This work addresses the challenges of weakly supervised video moment retrieval using only video-level labels, where existing methods suffer from low-quality temporal proposals, difficulty in distinguishing intra-video negative samples, and training instability. To overcome these limitations, the authors propose a multi-task training framework that generates multiple temporal proposals and integrates learnable Gaussian masks to construct high-quality positive samples. Additionally, they introduce an easy/hard negative sampling strategy and incorporate both forward and inverse masked query reconstruction tasks. This approach enhances the quality of positive sample representations and improves model robustness, effectively mitigating the reliance on single auxiliary tasks or suboptimal proposals prevalent in prior work. Experimental results demonstrate significant performance gains over current weakly supervised methods on two standard benchmarks.

Abstract

This study focuses on weakly-supervised Video Moment Retrieval (VMR), aiming to identify a moment semantically similar to the given query within an untrimmed video using only video-level correspondences, without relying on temporal annotations during training. Previous methods either aggregate predictions for all instances in the video, or indirectly address the task by proposing reconstructions for the query. However, these methods often produce low-quality temporal proposals, struggle with distinguishing misaligned moments in the same video, or lack stability due to a reliance on a single auxiliary task. To address these limitations, we present a novel weakly-supervised method called Multi-proposal Collaboration and Multi-task Training (MCMT). Initially, we generate multiple proposals and derive corresponding learnable Gaussian masks from them. These masks are then combined to create a high-quality positive sample mask, highlighting video clips most relevant to the query. Concurrently, we classify other clips in the same video as the easy negative sample and the entire video as the hard negative sample. During training, we introduce forward and inverse masked query reconstruction tasks to impose more substantial constraints on the network, promoting more robust and stable retrieval performance. Extensive experiments on two standard benchmarks affirm the effectiveness of the proposed method in VMR.

Problem

• Weakly-supervised Learning
• Video Moment Retrieval
• Temporal Localization
• Multi-task Training
• Proposal Generation

Innovation

• Multi-proposal Collaboration
• Multi-task Training
• Weakly-supervised Learning
• Video Moment Retrieval
• Masked Query Reconstruction