Weaver: End-to-End Agentic System Training for Video Interleaved Reasoning

2026-02-05

AI Summary

Existing video reasoning approaches struggle to effectively handle complex multimodal information in long videos due to representation mismatches and limited perceptual capabilities. This work proposes the first end-to-end trainable multimodal reasoning agent, which dynamically invokes external tools to progressively acquire critical visual cues. The agent autonomously explores effective tool-combination strategies through reinforcement learning without trajectory-level supervision. By integrating multimodal understanding, dynamic decision-making, and end-to-end optimization, the proposed system substantially outperforms current methods across multiple challenging video reasoning benchmarks, achieving particularly notable performance gains on long-video tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Reasoning with Beliefs

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Video reasoning constitutes a comprehensive assessment of a model's capabilities, as it demands robust perceptual and interpretive skills, thereby serving as a means to explore the boundaries of model performance. While recent research has leveraged text-centric Chain-of-Thought reasoning to augment these capabilities, such approaches frequently suffer from representational mismatch and restricted by limited perceptual acuity. To address these limitations, we propose Weaver, a novel, end-to-end trainable multimodal reasoning agentic system. Weaver empowers its policy model to dynamically invoke diverse tools throughout the reasoning process, enabling progressive acquisition of crucial visual cues and construction of authentic multimodal reasoning trajectories. Furthermore, we integrate a reinforcement learning algorithm to allow the system to freely explore strategies for employing and combining these tools with trajectory-free data. Extensive experiments demonstrate that our system, Weaver, enhances performance on several complex video reasoning benchmarks, particularly those involving long videos.

Problem

• video reasoning
• representational mismatch
• perceptual acuity
• multimodal reasoning
• long videos

Innovation

• multimodal reasoning
• end-to-end trainable
• agentic system
• reinforcement learning
• video interleaved reasoning