Rethinking Weakly-supervised Video Temporal Grounding From a Game Perspective

2026-05-25

AI Summary

Weakly supervised video temporal grounding faces two major challenges: coarse-grained cross-modal alignment and reliance on complex moment proposals, which lead to inaccurate boundary localization and low efficiency. This work introduces cooperative game theory for the first time to model video frames and query words as game-theoretic agents. By leveraging multi-granularity cross-modal interactions, the method learns their uncertain correspondences and employs a query-guided frame-level scoring mechanism to directly localize target moments, eliminating the need for conventional proposal generation. This approach achieves fine-grained cross-modal alignment, significantly improving both localization accuracy and computational efficiency, and establishes state-of-the-art performance on Charades-STA and ActivityNet Caption benchmarks.

Abstract

This paper addresses the challenging task of weakly-supervised video temporal grounding. Existing approaches are generally based on the moment proposal selection framework that utilizes contrastive learning and reconstruction paradigm for scoring the pre-defined moment proposals. Although they have achieved significant progress, we argue that their current frameworks have overlooked two indispensable issues: 1) Coarse-grained cross-modal learning: previous methods solely capture the global video-level alignment with the query, failing to model the detailed consistency between video frames and query words for accurately grounding the moment boundaries. 2) Complex moment proposals: their performance severely relies on the quality of proposals, which are also time-consuming and complicated for selection. To this end, in this paper, we make the first attempt to tackle this task from a novel game perspective, which effectively learns the uncertain relationship between each vision-language pair with diverse granularity and flexible combination for multi-level cross-modal interaction.Specifically, we creatively model each video frame and query word as game players with multivariate cooperative game theory to learn their contribution to the cross-modal similarity score. By quantifying the trend of frame-word cooperation within a coalition via the game-theoretic interaction, we are able to value all uncertain but possible correspondence between frames and words. Finally, instead of using moment proposals, we utilize the learned query-guided frame-wise scores for better moment localization.Experiments show that our method achieves superior performance on both Charades-STA and ActivityNet Caption datasets.

Problem

• weakly-supervised video temporal grounding
• coarse-grained cross-modal learning
• moment proposal
• frame-word alignment
• temporal localization

Innovation

• cooperative game theory
• weakly-supervised temporal grounding
• cross-modal interaction
• frame-word alignment
• proposal-free localization