CoFiE: Coarse-to-Fine Evidence Selection for Efficient Streaming Video Understanding

2026-09-03

AI Summary

本文提出CoFiE框架，通过粗到细的证据选择方法，在视觉编码前过滤帧，并在语言模型预填充时进行细化，有效减少了流视频理解中的延迟并提高了准确性。

Abstract

Streaming video understanding requires Vision Language Models (VLLMs) to process growing video streams and answer user questions under tight latency constraints. Existing methods improve efficiency through token pruning and memory-bank schemes, but mainly reduce visual tokens after visual encoding. Consequently, downstream token pruning alone cannot substantially reduce end-to-end latency because the expensive frame encoding cost has already been incurred. We propose CoFiE, a Coarse-to-Fine Evidence Selection framework that decouples evidence selection into a coarse, query-agnostic filtering stage before the vision encoder and a fine, query-specific refinement stage during LLM prefill. CoFiE introduces Novelty-Guided Frame Filtering to retain visually distinctive candidate frames and Query-Specific Evidence Refinement to select the frames most relevant to the user query. This design removes substantial redundancy before frame encoding while preserving query-specific refinement once semantic information becomes available. Experiments show that CoFiE establishes a new state-of-the-art accuracy-efficiency trade-off across multiple video understanding benchmarks, reaching 78.86% accuracy on StreamingBench and 68.72% on OvO-Bench, with improvements of up to 3.15% over prior methods. Even with up to 80% evidence-frame filtering, CoFiE outperforms strong open-source multimodal models while improving end-to-end inference latency by up to 2.54 times.

Problem

• Streaming Video Understanding
• End-to-End Latency
• Visual Encoding
• Token Pruning
• Efficiency

Innovation

• Coarse-to-Fine Evidence Selection
• Novelty-Guided Frame Filtering
• Query-Specific Evidence Refinement