MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

2026-03-10

AI Summary

This work proposes a novel task of multi-agent first-person video question answering, aimed at addressing the challenges of understanding and reasoning over long-duration videos captured synchronously from multiple embodied agents. To facilitate research in this direction, we introduce MA-EgoQA, the first systematic benchmark comprising 1.7k questions spanning social interactions, task collaboration, theory of mind, temporal reasoning, and environmental interaction, along with a formal definition of the multi-agent egocentric video joint understanding problem. We further present EgoMAS, a baseline model incorporating cross-agent shared memory and dynamic retrieval mechanisms. Experimental results demonstrate that existing methods achieve limited performance on this task, highlighting its inherent difficulty and establishing a foundation for future research.

Technology Category

• Multiagent Systems: Multiagent Planning
• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

As embodied models become powerful, humans will collaborate with multiple embodied AI agents at their workplace or home in the future. To ensure better communication between human users and the multi-agent system, it is crucial to interpret incoming information from agents in parallel and refer to the appropriate context for each query. Existing challenges include effectively compressing and communicating high volumes of individual sensory inputs in the form of video and correctly aggregating multiple egocentric videos to construct system-level memory. In this work, we first formally define a novel problem of understanding multiple long-horizon egocentric videos simultaneously collected from embodied agents. To facilitate research in this direction, we introduce MultiAgent-EgoQA (MA-EgoQA), a benchmark designed to systemically evaluate existing models in our scenario. MA-EgoQA provides 1.7k questions unique to multiple egocentric streams, spanning five categories: social interaction, task coordination, theory-of-mind, temporal reasoning, and environmental interaction. We further propose a simple baseline model for MA-EgoQA named EgoMAS, which leverages shared memory across embodied agents and agent-wise dynamic retrieval. Through comprehensive evaluation across diverse baselines and EgoMAS on MA-EgoQA, we find that current approaches are unable to effectively handle multiple egocentric streams, highlighting the need for future advances in system-level understanding across the agents. The code and benchmark are available at https://ma-egoqa.github.io.

Problem

• egocentric videos
• multiple embodied agents
• multi-agent understanding
• video question answering
• system-level memory

Innovation

• multi-agent egocentric video
• system-level memory
• embodied AI collaboration
• dynamic retrieval
• video question answering