Learning Situated Awareness in the Real World

2026-02-18

AI Summary

This work addresses a critical limitation in current vision-language foundation models: their inability to reason about observer-centric spatial relationships tied to viewpoint, pose, and motion. To bridge this gap, the authors introduce SAW-Bench, the first benchmark for observer-centered situational awareness in real-world settings, comprising 786 first-person videos captured with Ray-Ban Meta smart glasses and 2,071 human-annotated question-answer pairs spanning six embodied spatial reasoning tasks. Evaluations reveal that even the best-performing model (Gemini 3 Flash) lags behind human performance by 37.66%, exposing systematic deficiencies in reasoning about camera geometry and spatial consistency from an egocentric perspective.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over possible actions in context. However, most existing benchmarks for multimodal foundation models (MFMs) emphasize environment-centric spatial relations (relations among objects in a scene), while largely overlooking observer-centric relationships that require reasoning relative to agent's viewpoint, pose, and motion. To bridge this gap, we introduce SAW-Bench (Situated Awareness in the Real World), a novel benchmark for evaluating egocentric situated awareness using real-world videos. SAW-Bench comprises 786 self-recorded videos captured with Ray-Ban Meta (Gen 2) smart glasses spanning diverse indoor and outdoor environments, and over 2,071 human-annotated question-answer pairs. It probes a model's observer-centric understanding with six different awareness tasks. Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash. Beyond this gap, our in-depth analysis uncovers several notable findings; for example, while models can exploit partial geometric cues in egocentric videos, they often fail to infer a coherent camera geometry, leading to systematic spatial reasoning errors. We position SAW-Bench as a benchmark for situated spatial intelligence, moving beyond passive observation to understanding physically grounded, observer-centric dynamics.

Problem

• situated awareness
• egocentric perception
• observer-centric reasoning
• spatial understanding
• multimodal foundation models

Innovation

• situated awareness
• egocentric perception
• multimodal foundation models
• spatial reasoning
• real-world benchmark