PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

2026-06-23

AI Summary

This work addresses the challenge faced by multimodal large language models in accurately interpreting complex spatial relationships in pointing gesture–based visual grounding tasks. To this end, the authors propose PointVG-R, a novel approach that introduces a geometry-aware visual chain-of-thought mechanism to emulate human-like iterative reasoning. By integrating supervised fine-tuning with reinforcement learning and leveraging explicit geometric cues from images, PointVG-R achieves precise object localization. Key contributions include a pioneering geometry reasoning pipeline, the EgoPoint-CoT dataset for visual chain-of-thought reasoning, and a group-variance-based adaptive importance weighting strategy that dynamically refines reinforcement learning reward signals. Experimental results demonstrate that PointVG-R outperforms baseline methods by 15.86 percentage points in mIoU, establishing state-of-the-art performance, while ablation studies confirm the efficacy of each proposed component.

Abstract

Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by $\textbf{15.86}$ points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Code: https://github.com/lingli1724/PointVG-R.

Problem

• pointing-based visual grounding
• spatial relationships
• geometric reasoning
• multi-modal large language models
• visual chain of thought

Innovation

• geometric reasoning
• visual chain of thought
• pointing-based visual grounding
• reinforcement learning
• adaptive importance weighting