Beyond Single Object: Learning 3D Relations with Large Language Models

2026-08-16

AI Summary

This study addresses the deficiency of fine-grained multi-object comparison and geometric reasoning in existing 3D large models by proposing the Multi-3DLLM framework with a PIT interaction module. We construct the MO3D dataset and Mini-apps benchmark, alongside releasing the first 3D instruction set tailored for multi-object comparison. Through patch-level interaction mechanisms and hybrid data fine-tuning, our approach enables cross-object geometric relationship modeling and precise reasoning. Experimental results demonstrate that Multi-3DLLM comprehensively outperforms baselines on MO3D, exhibiting significant geometric reasoning capabilities. Furthermore, it shows effective positive transfer to single-object classification tasks, thereby bridging a critical research gap in multi-object 3D understanding.

Abstract

We address a fundamental gap in 3D-LLMs: existing models focus on single-object/scene description, struggling with detailed, inter-object comparison. We propose a framework for detailed object-level reasoning across multiple objects with three components: (1) MO3D (Multi-Object in 3D), an instruction dataset requiring fine-grained multi-object comparison; (2) Multi-3DLLM, using a minimal Patch-Interaction Transformer (PIT) that models inter-/intra-object relationships while preserving local geometry; (3) Mini-apps, two application-driven benchmarks (Shape Mating, Change Captioning) that probe geometric understanding for practical use. Recent 3D-LLMs and 2D-VLMs perform poorly on these tasks, lacking both comparison-centric design and geometric awareness. In contrast, Multi-3DLLM trained on our mixture data learns geometric reasoning, surpasses all baselines on MO3D, and provides positive transfer to single-object classification.

Problem

• 3D-LLMs
• Multi-object reasoning
• Inter-object comparison
• Geometric understanding

Innovation

• Multi-Object 3D Reasoning
• Patch-Interaction Transformer
• MO3D Dataset
• Geometric Understanding
• 3D Large Language Models