EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

2026-07-01

AI Summary

Current benchmarks struggle to independently evaluate agents’ ability to select actions based solely on egocentric perspectives in multi-agent scenarios. This work introduces a new capability—Ego-centric Action Selection (EAS)—and presents EgoGapBench, a diagnostic benchmark designed to isolate first-person visual inputs from confounding influences of others’ behaviors, thereby specifically assessing an agent’s capacity to make reasonable action decisions using only its own viewpoint. Experimental results show that humans perform robustly on this benchmark, whereas both open- and closed-source multimodal large language models exhibit substantially lower accuracy and frequently misattribute actions to themselves that are actually performed by others. Fine-tuning exclusively on EgoGapBench yields modest performance gains but remains far below human-level competence, revealing a systematic deficiency in current models’ EAS capabilities.

Abstract

Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when first-person body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent's perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance. These results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection.

Problem

• egocentric action selection
• multi-agent scenes
• first-person view
• perspective understanding
• action selection

Innovation

• Egocentric Action Selection
• EgoGapBench
• Multi-Agent Scenes
• First-Person View
• MLLM Evaluation