EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

2026-06-23

AI Summary

Current vision-language models lack a unified benchmark for evaluating multimodal temporal reasoning—spanning retrospective, online, and prospective understanding—in first-person streaming video. To address this gap, this work proposes EgoSAT, the first comprehensive evaluation framework tailored to egocentric video streams, integrating 165 hours of video (1,997 clips) and approximately 4,800 high-quality question-answer pairs. EgoSAT further introduces an assessment mechanism that explicitly accounts for question answerability. Experimental results reveal that existing models exhibit significant weaknesses in both prospective and retrospective reasoning and consistently suffer from poor confidence calibration—often being highly confident yet incorrect. These findings provide critical diagnostic insights and establish a foundational benchmark for advancing research in streaming egocentric vision-language understanding.

Abstract

We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of modern vision-language models (VLMs). The benchmark targets streaming interaction understanding, where video frames arrive sequentially and models must continuously interpret evolving visual context. EgoSAT unifies several previously distinct tasks within a single streaming framework. In this formulation, queries about completed events correspond to retrospective reasoning, queries about ongoing activities require online understanding, and queries about future actions involve prospective anticipation. This unified setting requires models to reason about the past, present, and future while operating under the constraint that only previously observed frames are available. EgoSAT contains 1,997 unique videos spanning 165 hours of egocentric footage and around 4,800 high-quality question-answer pairs, carefully designed to probe reasoning across varying temporal contexts. Using this benchmark, we evaluate a diverse set of both open-weight and closed-weight VLMs, providing a systematic assessment of their ability for streaming interaction understanding. By distinguishing answerability and conducting diagnostics on confidence of models, we find existing models not only struggle with prospective and retrospective modeling, but also exhibit severe mis-calibration: confidence often fails to track inherent answerability, leading to dangerous "confidently wrong" behaviors. Project page: https://leiyj23.github.io/EgoSAT/

Problem

• egocentric video
• streaming interaction understanding
• vision-language models
• temporal reasoning
• model calibration

Innovation

• egocentric video
• streaming reasoning
• vision-language models
• temporal understanding
• model calibration