TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs

2025-09-22

AI Summary

Existing reinforcement learning approaches for video temporal grounding suffer from on-policy sampling constraints, hindering efficient convergence in large search spaces and impeding precise temporal localization. This paper proposes TempSamp-R1, a reinforcement fine-tuning framework tailored for multimodal large language models. Its core contributions are threefold: (1) off-policy supervision guided by ground-truth annotations to provide accurate temporal guidance; (2) nonlinear soft advantage estimation to reduce reward variance; and (3) integration of a hybrid chain-of-thought training paradigm to support diverse temporal reasoning. TempSamp-R1 significantly enhances video–language alignment, achieving state-of-the-art performance on Charades-STA, ActivityNet Captions, and QVHighlights. Moreover, it demonstrates strong generalization under few-shot settings.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces TempSamp-R1, a new reinforcement fine-tuning framework designed to improve the effectiveness of adapting multimodal large language models (MLLMs) to video temporal grounding tasks. We reveal that existing reinforcement learning methods, such as Group Relative Policy Optimization (GRPO), rely on on-policy sampling for policy updates. However, in tasks with large temporal search spaces, this strategy becomes both inefficient and limited in performance, as it often fails to identify temporally accurate solutions. To address this limitation, TempSamp-R1 leverages ground-truth annotations as off-policy supervision to provide temporally precise guidance, effectively compensating for the sparsity and misalignment in on-policy solutions. To further stabilize training and reduce variance in reward-based updates, TempSamp-R1 provides a non-linear soft advantage computation method that dynamically reshapes the reward feedback via an asymmetric transformation. By employing a hybrid Chain-of-Thought (CoT) training paradigm, TempSamp-R1 optimizes a single unified model to support both CoT and non-CoT inference modes, enabling efficient handling of queries with varying reasoning complexity. Experimental results demonstrate that TempSamp-R1 outperforms GRPO-based baselines, establishing new state-of-the-art performance on benchmark datasets: Charades-STA (R1@0.7: 52.9%, +2.7%), ActivityNet Captions (R1@0.5: 56.0%, +5.3%), and QVHighlights (mAP: 30.0%, +3.0%). Moreover, TempSamp-R1 shows robust few-shot generalization capabilities under limited data. Code: https://github.com/HVision-NKU/TempSamp-R1

Problem

• Improving video temporal grounding in multimodal language models
• Addressing inefficiency of on-policy sampling in large temporal spaces
• Enhancing training stability with non-linear advantage computation

Innovation

• Uses ground-truth annotations for off-policy temporal guidance
• Implements non-linear soft advantage computation for reward stabilization
• Employs hybrid Chain-of-Thought training for unified model optimization