Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views

2026-06-22

AI Summary

This work addresses the challenges of inconsistent cross-view reasoning and fragile view selection in multi-view 3D visual question answering, which arise when relying solely on sparse answer-level supervision. To overcome these issues, the authors decouple the reasoning process into three stages: global map construction, question-guided view trajectory planning, and egocentric answer prediction. They introduce a dense, annotation-free reward mechanism that provides process-level supervision by combining global geometric consistency with local sequential view-selection rewards. Built upon frozen 3D vision foundation models (e.g., VGGT+SAM3), their approach incorporates pseudo-target generation, trajectory-level policy optimization (GRPO), and map-anchored learning. Experiments on MindCube, VSI-Bench, and BLINK (MV) demonstrate substantial improvements over strong multi-image baselines, validating the efficacy of dense process supervision.

Abstract

Multi-view 3D Visual Question Answering (MV3D-VQA) requires integrating partial observations into a coherent 3D scene representation and selecting informative viewpoints for multi-step spatial reasoning. However, current multimodal LLMs are typically trained with sparse, answer-level supervision, which often yields inconsistent cross-view reasoning and brittle view selection. We present DR-MV3D (Dense Reward for MV3D-VQA), a map-grounded learning framework that provides dense, verifiable rewards to supervise the reasoning process. Our approach decomposes MV3D-VQA into (i) allocentric global map construction, (ii) question-conditioned view-trajectory planning, and (iii) egocentric grounding for answer prediction. To make intermediate steps learnable without manual annotations, we introduce two rewards: a global consistency reward that aligns the predicted map with geometry-consistent pseudo targets from frozen 3D vision foundation models (e.g., VGGT + SAM3), and a local trajectory reward that supervises ordered viewpoint selection. We optimize the full pipeline with trajectory-level policy optimization (GRPO). Experiments on MindCube, VSI-Bench, and BLINK (MV) show that DR-MV3D consistently improves over strong multi-image baselines, supporting the effectiveness of process-level dense supervision for multi-view 3D reasoning.

Problem

• Multi-view 3D Visual Question Answering
• dense reward
• spatial reasoning
• view selection
• 3D scene representation

Innovation

• dense reward
• multi-view 3D reasoning
• global map construction
• view-trajectory planning
• trajectory-level policy optimization