Learning to Refuse: Refusal-Aware Reinforcement Fine-Tuning for Hard-Irrelevant Queries in Video Temporal Grounding

2025-11-28

AI Summary

Current video temporal grounding (VTG) models lack discriminative capability against semantically similar yet irrelevant “hard negative” queries, leading to spurious matches. To address this, we propose a rejection-aware reinforcement fine-tuning framework—the first to introduce a fine-grained rejection mechanism specifically designed for hard irrelevant queries. Our method builds upon the Group Relative Policy Optimization (GRPO) framework and incorporates four synergistic reward objectives: format consistency, rejection IoU, interpretability, and query refinement—jointly enhancing relevance discrimination and semantic reasoning. Evaluated on our newly constructed HI-VTG benchmark, the approach significantly improves both rejection accuracy and grounding precision. It demonstrates strong generalizability across diverse query distributions and model-agnostic compatibility, enabling seamless integration with various large vision-language models in VTG systems.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Image and Video Retrieval
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video Temporal Grounding (VTG) aims to localize a temporal segment in a video corresponding to a natural language query. However, existing VTG models assume that a relevant segment always exists, causing them to always predict a target segment even when the query is irrelevant to the video. While recent approaches attempt to handle irrelevant queries, they can only reject those that are entirely unrelated to the video and still fail to handle hard-irrelevant queries that are semantically similar but not actually relevant. To address this, we propose Refusal-Aware Reinforcement Fine-Tuning (RA-RFT) to effectively refuse hard-irrelevant queries in VTG. Our method is based on the Group Relative Policy Optimization (GRPO) framework and integrates four reward objectives-format, refuse-IoU, explain, and query correction-to improve both relevance discrimination and fine-grained semantic reasoning. In addition, to effectively support RA-RFT, we construct a Hard-Irrelevant VTG (HI-VTG) dataset, which includes hard-irrelevant queries and their refusal answers. We demonstrate the effectiveness of our method across various relevance-aware VTG scenarios, including hard-irrelevant VTG, simply-shuffled RA-VTG, and human-annotated RA-VTG settings. We also show that the proposed method is scalable by applying it to various LVLM-based VTG models. Our code is available at https://github.com/JINSUBY/RA-RFT.

Problem

• Addressing hard-irrelevant queries in Video Temporal Grounding that are semantically similar but not actually relevant
• Developing refusal-aware models to reject queries without corresponding video segments
• Improving relevance discrimination and semantic reasoning for irrelevant video queries

Innovation

• Refusal-Aware Reinforcement Fine-Tuning for hard-irrelevant queries
• Integrates four reward objectives to improve relevance discrimination
• Constructs a Hard-Irrelevant VTG dataset to support training