DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding

2026-07-01

AI Summary

This work addresses the performance bottleneck in zero-shot video temporal grounding caused by complex multi-stage queries, which existing methods struggle to handle due to limited temporal and causal reasoning capabilities. To overcome this, we propose a difficulty-adaptive dual-path reasoning framework that jointly performs keyframe selection and temporal reasoning difficulty estimation via a Determinantal Point Process (DPP). Spectral entropy is introduced as a difficulty metric to dynamically route samples to either a fast direct prediction path or a slow step-by-step reasoning path. Additionally, a Temporal Markup Prompting mechanism decomposes complex grounding tasks into structured sub-steps. Our approach achieves state-of-the-art zero-shot performance on Charades-STA and ActivityNet Captions, improving mIoU by up to 3.5 points while reducing the number of processed frames by over 7×.

Abstract

Zero-shot video temporal grounding (VTG) localizes events in untrimmed videos from natural language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure -- a disparity we call the reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured reasoning in large vision-language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.

Problem

• zero-shot video temporal grounding
• reasoning gap
• complex multi-stage queries
• temporal ordering
• causal structure

Innovation

• Difficulty-Adaptive Routing
• Zero-Shot Video Temporal Grounding
• Determinantal Point Process
• Temporal Markup Prompting
• Structured Reasoning