video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

2026-06-23

AI Summary

Current video large language models often sacrifice critical information by reducing frame rates or spatial resolution due to computational and memory constraints, leading to degraded question-answering accuracy. To address this, this work proposes an end-to-end two-stage video understanding framework that first coarsely localizes relevant segments and then revisits them with high fidelity. The approach jointly optimizes mechanisms for rewatching, rephrasing questions, and regenerating answers. Notably, it achieves the first end-to-end rewatching without requiring chain-of-thought cold-start prompting, mitigates the “answer-first, think-later” bias inherent in pretrained models through answer regeneration, and enhances question-video alignment via question refinement. Experiments demonstrate that the method consistently outperforms existing baselines and rewatching strategies while significantly reducing computational overhead, thereby improving video question-answering accuracy.

Abstract

Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to miss critical information for question answering (QA). A practical and efficient solution is a two-stage paradigm: first perform coarse video understanding to localize relevant segments, and then re-watch these segments at higher temporal or spatial fidelity. In this paper, we present video-SALMONN-R$^3$, the first end-to-end video-LLM that enables re-watch through reinforcement learning without relying on chain-of-thought (CoT) cold-start. This design removes the need for costly CoT data annotations and avoids CoT-based supervised fine-tuning (SFT), which can otherwise degrade the pretrained video understanding abilities. To address the mismatch between the reasoning-first behavior induced by re-watch and the answer-first tendency of pretrained video-LLMs, we propose a re-answer strategy, in which the model first produces a direct answer in the first watch and then refines it after re-watching. Finally, to improve question adherence during re-watching, we propose a re-ask mechanism that re-injects the query when revisiting localized segments. Experimental results show that video-SALMONN-R$^3$ consistently outperforms both the base model and the QA-SFT baseline, while surpassing prior re-watch-based approaches with significantly lower computational cost. Code, models, and data will be publicly released upon acceptance.

Problem

• video understanding
• large language models
• question answering
• computational efficiency
• information loss

Innovation

• reinforcement learning
• re-watch
• re-answer
• re-ask
• video large language model