EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

2026-06-30

AI Summary

Current binary safety evaluations struggle to distinguish genuinely hazardous actions from superficially suspicious yet benign everyday behaviors, leading embodied agents’ safety guardrails to suffer from false positives or missed detections. This work proposes SafeEgo, the first fine-grained safety benchmark, comprising 1,200 first-person videos structured as contrastive pairs that differ only by a single critical factor, thereby precisely probing whether models rely on decisive cues rather than coarse scene categories. The benchmark features dual tracks—contextual understanding and textual interference—and incorporates sub-second temporal annotations to evaluate both open- and closed-source vision-language models. Experiments reveal that while existing models recognize dangerous scenarios in general, they frequently miss the precise moments of risk, particularly for context-dependent hazards, and are highly susceptible to misleading textual prompts, resulting in either excessive intervention or severe oversight.

Abstract

Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genuinely unsafe situations while avoiding unnecessary intervention on routine but superficially alarming activity, a distinction that binary safety benchmarks obscure. We introduce EgoSafetyBench, an egocentric video benchmark of 1,200 robot-view scenarios annotated at half-second granularity, to evaluate VLMs as streaming guards across two tracks. The situational track (800 scenarios) spans four families, from routine and safe-but-suspicious scenes to obvious and contextual hazards. The visual-channel track (400 scenarios) targets in-scene text-a sign, sticker, or label visible in the scene-that can misrepresent the physical situation, pairing each misleading sign with a truthful version to test both whether a guard flags the text as misleading and whether the text corrupts its physical-safety judgment. Both tracks use contrastive ladders: near-identical scenarios differing only in a single visible deciding cue, so a correct call must hinge on that cue rather than the overall scene type. We evaluate ten open- and closed-source VLMs. We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments, particularly contextual hazards. Furthermore, misleading in-scene signs degrade all tested guards: vulnerable models miss up to a third of hazards, while robust models over-intervene on safe content. Matched controls reveal that apparent safety robustness often reflects indiscriminate alarming rather than true physical reasoning.

Problem

• embodied VLMs
• runtime safety
• egocentric video benchmark
• contextual hazards
• in-scene text

Innovation

• Egocentric Video Benchmark
• Embodied VLMs
• Runtime Safety Guard
• Contrastive Ladders
• In-scene Text Robustness