Problem
Research questions and friction points this paper is trying to address.
long-video understanding
context construction
frozen vision-language model
recursive search
executable program
VideoHarness-RSI: Recursive Harness Self-Improvement for Long-Video Understanding with Frozen Vision-Language Models
🤖 AI Summary
本文通过VIDEOHARNESS-RSI方法，优化长视频理解中模型上下文构建程序，提升冻结视觉-语言模型的表现，超越手动设计的基线。
📝 Abstract
Long-video understanding depends critically on how a limited model context is constructed from a much longer video. Existing approaches improve this process through compression, retrieval, memory, and agentic evidence acquisition, but these mechanisms are typically introduced as part of a manually designed inference system or optimized together with other components. This makes it difficult to isolate a simpler question: how much can be gained by improving the executable context-construction program alone? We study this question through VIDEOHARNESS-RSI, a controlled baseline for recursively searching executable context constructors around a frozen vision-language model (VLM). An outer-loop proposer uses prior programs, evaluation outcomes, and execution traces to generate candidate harnesses, which are executed and evaluated end to end before successful variants are retained for further search. This makes long-video understanding a controlled instance of automated harness design: the searchable object is executable program structure, while the answering model and interface remain fixed. Starting from uniform sampling, recursive harness search consistently finds room for improvement and surpasses several weaker hand-crafted baselines. Starting instead from a stronger hand-crafted baseline, the same RSI process yields a further improvement. The selected harness also transfers to additional long-video benchmarks without further search. Together, these results establish executable context construction as a distinct optimization layer and provide a reproducible baseline for studying harness discovery and transfer around frozen VLMs.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Recursive Harness Self-Improvement
Executable Context Construction
Frozen Vision-Language Models
Automated Harness Design
🔎 Similar Papers
No similar papers found.
