Native Active Perception as Reasoning for Omni-Modal Understanding

2026-06-17

AI Summary

This work addresses the inefficiency of conventional video understanding models, whose inference complexity scales linearly with video duration due to their “watch-all” paradigm, rendering them ill-suited for long-form content. To overcome this limitation, the authors propose OmniAgent—the first natively multimodal agent that formulates video understanding as a partially observable Markov decision process (POMDP). OmniAgent operates through an iterative observe-think-act loop, dynamically selecting actions to extract salient audiovisual cues and record them into persistent textual memory, thereby decoupling inference complexity from video length. The approach leverages a two-stage supervised fine-tuning strategy with trajectory synthesis and the TAURA reinforcement learning algorithm, which performs credit assignment based on episode-level entropy. Evaluated across ten benchmarks—including VideoMME and LVBench—OmniAgent achieves state-of-the-art open-source performance, with its 7B variant outperforming Qwen2.5-VL-72B (47.3%) by attaining 50.5% on LVBench.

Abstract

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

Problem

• video understanding
• computational cost
• long video
• context scaling
• passive perception

Innovation

• Active Perception
• Omni-Modal Agent
• POMDP
• TAURA
• Test-Time Scaling