HiCrew: Hierarchical Reasoning for Long-Form Video Understanding via Question-Aware Multi-Agent Collaboration

2026-04-23

AI Summary

Long-form video understanding faces challenges from spatiotemporal redundancy and cross-segment narrative dependencies. Existing approaches often compromise temporal coherence when compressing visual information, and multi-agent systems typically lack adaptability to question intent. This work proposes HiCrew, a hierarchical multi-agent collaboration framework that preserves temporal topology through shot-boundary detection and semantic clustering to construct a hybrid tree structure. It introduces a question-aware captioning mechanism to generate precise semantic descriptions and incorporates a planning layer that dynamically allocates agent roles and execution paths based on question complexity. HiCrew significantly enhances causal and temporal reasoning capabilities, achieving state-of-the-art performance on the EgoSchema and NExT-QA benchmarks, with particularly notable gains in temporal and causal reasoning tasks.

Technology Category

• Multiagent Systems: Teamwork
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Video Understanding & Activity Analysis

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Long-form video understanding remains fundamentally challenged by pervasive spatiotemporal redundancy and intricate narrative dependencies that span extended temporal horizons. While recent structured representations compress visual information effectively, they frequently sacrifice temporal coherence, which is critical for causal reasoning. Meanwhile, existing multi-agent frameworks operate through rigid, pre-defined workflows that fail to adapt their reasoning strategies to question-specific demands. In this paper, we introduce HiCrew, a hierarchical multi-agent framework that addresses these limitations through three core contributions. First, we propose a Hybrid Tree structure that leverages shot boundary detection to preserve temporal topology while performing relevance-guided hierarchical clustering within semantically coherent segments. Second, we develop a Question-Aware Captioning mechanism that synthesizes intent-driven visual prompts to generate precision-oriented semantic descriptions. Third, we integrate a Planning Layer that dynamically orchestrates agent collaboration by adaptively selecting roles and execution paths based on question complexity. Extensive experiments on EgoSchema and NExT-QA validate the effectiveness of our approach, demonstrating strong performance across diverse question types with particularly pronounced gains in temporal and causal reasoning tasks that benefit from our hierarchical structure-preserving design.

Problem

• long-form video understanding
• temporal coherence
• narrative dependencies
• multi-agent reasoning
• spatiotemporal redundancy

Innovation

• Hierarchical Reasoning
• Multi-Agent Collaboration
• Question-Aware Captioning
• Temporal Coherence
• Hybrid Tree Structure