EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

2026-05-26

AI Summary

Existing benchmarks struggle to jointly evaluate the multimodal perception, multi-hop tool use, and dynamic human–agent interaction capabilities required by AI agents in open-ended environments. To address this gap, this work introduces an interactive multimodal benchmark grounded in first-person videos, encompassing 1,045 everyday tasks within user–agent–tool interaction scenarios. The benchmark enforces the integration of visual perception and tool-augmented multi-hop reasoning through a three-stage collaborative reasoning pipeline and employs multi-agent simulations to generate high-fidelity user feedback. It presents the first framework capable of jointly assessing these three core competencies, featuring a deterministic joint validation protocol and a dual-dimensional evaluation mechanism that considers both process and outcome equivalence. Evaluations of eight state-of-the-art video-based multimodal large language models across four everyday scenarios reveal a best accuracy of only 30.62% (average: 19.43%), highlighting significant capability bottlenecks in current agents.

Abstract

As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop reasoning, and dynamic interaction with users. However, existing benchmarks fail to jointly evaluate these capabilities due to challenges in designing strictly coupled multi-capability tasks, simulating natural and task-constrained user feedback, and ensuring objective evaluation of dynamic interaction. To bridge this gap, we introduce EgoBench, the first interactive multimodal benchmark for tool-using agents. EgoBench comprises 1,045 egocentric-video-grounded tasks covering four daily scenarios, along with a user-agent-tool interactive environment for evaluation. We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning. We additionally develop a multi-agent simulated user within EgoBench to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses to agents. Furthermore, we establish a deterministic joint validation framework that guarantees objective assessment through process-based and result-based equivalence. Benchmarking eight SOTA video-MLLM agents on EgoBench reveals a severe performance ceiling: the best model achieves only 30.62% accuracy in the best-performing scenario, averaging 19.43% across all four scenarios. Finally, we conduct a multi-dimensional error analysis to disentangle failure modes, exposing capability bottlenecks for advancing future AI agents.

Problem

• multimodal perception
• tool-using agents
• dynamic interaction
• benchmark evaluation
• egocentric video

Innovation

• interactive benchmark
• egocentric video
• tool-using agents
• multimodal reasoning
• simulated user