AHA -- Predicting What Matters Next: Online Highlight Detection Without Looking Ahead

2025-09-19

AI Summary

Existing video highlight detection methods require full-video input, rendering them unsuitable for online streaming scenarios. This paper proposes Aha, the first online highlight detection framework enabling constant-memory inference over arbitrarily long video streams without access to future frames. Its core innovations include an autoregressive modeling paradigm and a novel Dynamic SinkCache mechanism, coupled with a lightweight, decoupled multi-task head that synchronously estimates frame-level relevance, informativeness, and uncertainty. Trained on large-scale human-annotated video data and grounded in multimodal vision-language representations, Aha achieves state-of-the-art performance: it surpasses the best offline methods by +5.9% and +8.3% mAP on TVSum and Mr. Hisum, respectively. These gains significantly enhance long-horizon understanding capabilities critical for real-time decision-making applications such as autonomous driving and search-and-rescue robotics.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Real-time understanding of continuous video streams is essential for intelligent agents operating in high-stakes environments, including autonomous vehicles, surveillance drones, and disaster response robots. Yet, most existing video understanding and highlight detection methods assume access to the entire video during inference, making them unsuitable for online or streaming scenarios. In particular, current models optimize for offline summarization, failing to support step-by-step reasoning needed for real-time decision-making. We introduce Aha, an autoregressive highlight detection framework that predicts the relevance of each video frame against a task described in natural language. Without accessing future video frames, Aha utilizes a multimodal vision-language model and lightweight, decoupled heads trained on a large, curated dataset of human-centric video labels. To enable scalability, we introduce the Dynamic SinkCache mechanism that achieves constant memory usage across infinite-length streams without degrading performance on standard benchmarks. This encourages the hidden representation to capture high-level task objectives, enabling effective frame-level rankings for informativeness, relevance, and uncertainty with respect to the natural language task. Aha achieves state-of-the-art (SOTA) performance on highlight detection benchmarks, surpassing even prior offline, full-context approaches and video-language models by +5.9% on TVSum and +8.3% on Mr.Hisum in mAP (mean Average Precision). We explore Aha's potential for real-world robotics applications given a task-oriented natural language input and a continuous, robot-centric video. Both experiments demonstrate Aha's potential effectiveness as a real-time reasoning module for downstream planning and long-horizon understanding.

Problem

• Detecting video highlights in real-time without future frame access
• Enabling online reasoning for continuous video streams in robotics
• Overcoming limitations of offline methods requiring complete video input

Innovation

• Autoregressive framework predicts frame relevance without future frames
• Dynamic SinkCache enables constant memory for infinite streams
• Multimodal vision-language model with lightweight decoupled heads