Agentic Collaborative Cognition for Zero-Shot 3D Understanding

2026-06-23

AI Summary

Existing zero-shot 3D understanding methods based on video keyframes are constrained by limited viewpoints and implicit 3D perception, leading to performance bottlenecks. This work proposes a dual-agent collaborative framework: a planning agent generates task-relevant novel viewpoints, while a perception agent constructs a structured, explicit 3D cognitive map. Through closed-loop iterative refinement, the framework fuses multi-view information and filters candidate objects. By integrating multimodal large language models, viewpoint planning, cross-view instance consistency labeling, and an iterative feedback mechanism, the approach overcomes reliance on limited perspectives and achieves more comprehensive and consistent 3D understanding. It attains state-of-the-art performance across six benchmarks, improving ScanRefer accuracy by 11.1%, BLEU-1 for 3D dialogue by 14.6, and SQA3D exact match (EM) by 2.1.

Abstract

Recent advancements have explored agentic zero-shot 3D understanding by reformulating it as video keyframe understanding with Multimodal Large Language Models (MLLMs). However, existing methods face an intrinsic bottleneck due to the finite observation perspectives inherent in videos and the implicit perception of 3D scenes. In this paper, we propose a collaborative multi-agent framework that assigns a Planning Agent to handle high-level viewpoint planning and supplement novel perspectives, and a Perception Agent to explicitly summarize the 3D scene into a structured holistic cognitive map. Specifically, Planning Agent first analyzes this cognitive map to determine query-relevant viewpoints and supplements missing critical perspectives to ensure comprehensive observation. Subsequently, Perception Agent documents object-level attributes from these views by assigning consistent instance identifiers across viewpoints, thereby integrating fragmented observations into the holistic cognitive map. In parallel, it provides feedback to filter out mismatched candidate objects and guide subsequent viewpoint planning. Through this closed-loop iterative process, two agents collaboratively figure out candidates until Perception Agent determines that sufficient information has been captured to complete the task. Extensive experiments demonstrate that our method achieves state-of-the-art performance on 6 benchmarks, with improvements of 11.1\% Acc@0.5 on ScanRefer, 14.6 BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D.

Problem

• zero-shot 3D understanding
• limited observation perspectives
• implicit 3D perception
• multimodal large language models
• 3D scene understanding

Innovation

• Agentic Collaboration
• Zero-Shot 3D Understanding
• Cognitive Map
• Viewpoint Planning
• Multimodal Large Language Models