HORNet: Task-Guided Frame Selection for Video Question Answering with Vision-Language Models

2026-03-19

AI Summary

Existing frame sampling strategies in video question answering often fail to adapt to downstream tasks, limiting both the efficiency and performance of vision-language models (VLMs). To address this, we propose HORNet—a lightweight, task-guided dynamic frame selection method that employs Group Relative Policy Optimization (GRPO) to train a compact policy network (<1M parameters) for selecting key frames for a frozen VLM. Built upon the newly introduced Select Any Frames (SAF) paradigm, HORNet decouples frame selection from inference, enabling cross-model transfer without retraining. Evaluated across six benchmarks, HORNet achieves 93% inference speedup using only 1% of the original frames, improves F1 by 1.7% on MSVD-QA, outperforms uniform sampling by 7.3 points on NExT-QA, and yields an 8.5% relative gain when applied to stronger VLMs.

Technology Category

• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Video question answering (VQA) with vision-language models (VLMs) depends critically on which frames are selected from the input video, yet most systems rely on uniform or heuristic sampling that cannot be optimized for downstream answering quality. We introduce \textbf{HORNet}, a lightweight frame selection policy trained with Group Relative Policy Optimization (GRPO) to learn which frames a frozen VLM needs to answer questions correctly. With fewer than 1M trainable parameters, HORNet reduces input frames by up to 99\% and VLM processing time by up to 93\%, while improving answer quality on short-form benchmarks (+1.7\% F1 on MSVD-QA) and achieving strong performance on temporal reasoning tasks (+7.3 points over uniform sampling on NExT-QA). We formalize this as Select Any Frames (SAF), a task that decouples visual input curation from VLM reasoning, and show that GRPO-trained selection generalizes better out-of-distribution than supervised and PPO alternatives. HORNet's policy further transfers across VLM answerers without retraining, yielding an additional 8.5\% relative gain when paired with a stronger model. Evaluated across six benchmarks spanning 341,877 QA pairs and 114.2 hours of video, our results demonstrate that optimizing \emph{what} a VLM sees is a practical and complementary alternative to optimizing what it generates while improving efficiency. Code is available at https://github.com/ostadabbas/HORNet.

Problem

• video question answering
• frame selection
• vision-language models
• temporal reasoning
• input efficiency

Innovation

• frame selection
• vision-language models
• video question answering
• policy optimization
• input efficiency