Problem
Research questions and friction points this paper is trying to address.
Streaming Video
Online Frame Selection
Query-Agnostic
Video Memory
Keyframe
SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selection
🤖 AI Summary
本文提出SVMemAgent，通过在线维护代表性视频帧记忆解决流媒体场景下的在线关键帧选择问题，使用GRPO训练以适应未知查询。
📝 Abstract
Most keyframe selection studies focus on offline settings, assuming access to the full video and query in advance. In contrast, real-world streaming scenarios require online frame selection under unknown video duration, without access to either the query or future frames during selection. To address this, we introduce Streaming Video Memory (SVMem), a compact and representative memory of previously observed content, updated continuously as the video stream unfolds. Building on this setting, we propose the Streaming Video Memory Agent (SVMemAgent), which dynamically maintains a memory by deciding at each timestep whether to replace an existing memory frame with the incoming frame or discard it. SVMemAgent is trained using Group Relative Policy Optimization (GRPO) with task-driven rewards derived from diverse question-answer pairs, implicitly exposing the policy to a distribution of queries during training so that SVMem retains generally informative frames at inference, when queries are unavailable. Experiments on both online and offline video benchmarks show that SVMemAgent consistently outperforms online frame selection baselines and achieves competitive performance with offline methods that assume access to the full video and query. Through task-driven rewards, SVMemAgent learns an emergent keyframe selection policy that prefers frames containing textual information, which may benefit downstream VideoQA tasks.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Streaming Video Memory Agent
Online Frame Selection
Group Relative Policy Optimization
Task-driven Rewards
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
