Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision

2026-04-23

AI Summary

This work addresses the challenge that multimodal large language models struggle to accurately interpret the spatial semantics of pointing gestures from a first-person perspective, often succumbing to spurious correlations such as visual proximity and exhibiting "pointing representation hallucination." To tackle this issue, the authors introduce EgoPoint-Bench, the first comprehensive benchmark for first-person pointing reasoning, comprising over 11,000 high-fidelity simulated and real-world samples spanning five evaluation dimensions and three levels of referential complexity. By integrating synthetic data fine-tuning with spatial-aware supervision, models trained using this approach significantly outperform existing open- and closed-source systems on the benchmark and demonstrate robust sim-to-real transferability.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Egocentric AI agents, such as smart glasses, rely on pointing gestures to resolve referential ambiguities in natural language commands. However, despite advancements in Multimodal Large Language Models (MLLMs), current systems often fail to precisely ground the spatial semantics of pointing. Instead, they rely on spurious correlations with visual proximity or object saliency, a phenomenon we term "Referential Hallucination." To address this gap, we introduce EgoPoint-Bench, a comprehensive question-answering benchmark designed to evaluate and enhance multimodal pointing reasoning in egocentric views. Comprising over 11k high-fidelity simulated and real-world samples, the benchmark spans five evaluation dimensions and three levels of referential complexity. Extensive experiments demonstrate that while state-of-the-art proprietary and open-source models struggle with egocentric pointing, models fine-tuned on our synthetic data achieve significant performance gains and robust sim-to-real generalization. This work highlights the importance of spatially aware supervision and offers a scalable path toward precise egocentric AI assistants. Project page: https://guyyyug.github.io/EgoPoint-Bench/

Problem

• Referential Reasoning
• Egocentric Vision
• Pointing Gesture
• Multimodal Large Language Models
• Spatial Semantics

Innovation

• Referential Hallucination
• Egocentric Vision
• Pointing Gesture
• Multimodal Large Language Models
• Sim-to-Real Generalization