Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation

2026-07-15

AI Summary

Existing multimodal large language models lack effective spatial perception and reasoning capabilities for egocentric visual question answering tasks. To address this limitation, this work proposes the Ego Scene Augmentation (ESA) framework, which introduces the Ego-element Graph as a structured intermediate representation. ESA leverages a vision foundation model to extract and integrate spatial elements and their relationships from first-person viewpoints, explicitly incorporating this structured spatial knowledge into the reasoning pipeline of multimodal large language models to actively enhance their spatial understanding. Experimental results demonstrate that ESA significantly improves performance on the EgoTextVQA benchmark, achieving accuracy gains of 8.14% and 8.72% in indoor and outdoor scenes, respectively, with particularly notable improvements observed in the shopping subset.

Abstract

Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world. However, existing MLLMs struggle to perform effective spatial reasoning in complex egocentric scenes due to their limited spatial perception capabilities. To this end, we introduce Ego Scene Augmentation (ESA), an egocentric spatial perception framework, which actively enhances the spatial perception capabilities from the egocentric perspective, powered by the proposed Ego-element Graph. Our core insight is leveraging the Ego-element Graph as an intermediary representation to augment the egocentric spatial perception of MLLMs via visual foundational models. Specifically, we 1) construct the Ego-element Graph, which encapsulates and integrates egocentric spatial features enabled by visual foundational models; 2) enhance the spatial perception capabilities of MLLMs via the Ego-element Graph for ego-perspective scenes. Our proposed ESA framework presents significant performance improvement on the EgoTextVQA benchmark. We achieve an 8.14% gain on the indoor setting and an 8.72% gain on the outdoor setting. Furthermore, our ESA shows the most impressive performance improvement in the shopping subset of the indoor setting. The project code is publicly available.

Problem

• Egocentric Spatial Perception
• Multimodal Large Language Models
• Spatial Reasoning
• Visual Question Answering
• Egocentric Scenes

Innovation

• Ego Scene Augmentation
• Ego-element Graph
• Egocentric Spatial Perception
• Multimodal Large Language Models
• Visual Question Answering