MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

2026-07-02

AI Summary

Existing medical video benchmarks focus solely on answer correctness while neglecting decision timing, thus failing to evaluate models’ real-time clinical judgment and early warning capabilities. To address this gap, this work introduces the first time-aware benchmark for medical video understanding, integrating 22 datasets and 5,419 question-answer instances across four temporal scenarios: retrospective, immediate, future-oriented, and proactive. The benchmark incorporates time-constrained evidence windows, a streaming inference framework, and an active monitoring setup that requires models to determine whether and when to trigger alerts under limited information. Time-aware evaluation metrics—such as responsiveness and post-decision stability—effectively bridge the gap between offline assessment and real-world deployment. Experiments reveal a significant performance drop in current multimodal models under streaming and proactive conditions, highlighting their limitations in time-sensitive decision-making.

Abstract

Existing medical video benchmarks primarily evaluate whether a model produces the correct answer, but rarely assess whether it answers at the right time. In real clinical settings, AI systems must decide not only what to predict, but also when to answer, defer judgment, or proactively raise alerts. This creates a critical gap between benchmark evaluation and deployment requirements. We present MedStreamBench, a benchmark for time-aware medical video understanding. MedStreamBench integrates 22 medical datasets and 5,419 QA instances across four temporal settings: retrospective, present, future, and proactive. Unlike conventional benchmarks that assume full-video access, MedStreamBench restricts models to temporally bounded evidence windows and supports both single-turn and streaming evaluation. We further introduce a proactive monitoring setting that requires models to determine whether and when clinically relevant alerts should be triggered. Beyond answer correctness, MedStreamBench evaluates temporal behavior through responsiveness and post-evidence stability. Experiments on leading general-purpose and medical vision-language models reveal a substantial gap between offline recognition and temporally grounded decision-making, with performance dropping markedly in streaming and proactive settings. Our benchmark is available at https://huggingface.co/datasets/Venn2024/MedStreamBench.

Problem

• medical video understanding
• time-aware evaluation
• streaming benchmark
• proactive alerting
• temporal decision-making

Innovation

• time-aware evaluation
• streaming medical video understanding
• proactive alerting
• temporal evidence window
• vision-language models