Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

2026-05-19

AI Summary

This work addresses the challenge of generalizing multimodal large language models to diverse camera setups in 3D localization tasks, where ambiguity in camera intrinsics hinders performance. To overcome this limitation, the authors propose an equation-anchored tool-use framework that explicitly integrates the pinhole camera model with depth tool outputs through geometric equations into the multimodal reasoning process. Specifically, spatial geometric relationships are constructed within the chain-of-thought reasoning, leveraging tool-derived depth estimates as back-projection variables to regress a 9-degree-of-freedom bounding box. Evaluated under strong perturbations involving camera intrinsic scaling from 0.5× to 1.5×, the method significantly outperforms RGB-only and existing tool-augmented baselines, demonstrating notably enhanced robustness to camera variations and superior localization accuracy—particularly in out-of-distribution scenarios.

Abstract

3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation $\hat{X} = (u_c - c_x)\bar{Z}/f_x$ explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from $0.5\times$ to $1.5\times$, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.

Problem

• 3D localization
• camera intrinsic ambiguity
• Multimodal Large Language Models
• 3D object detection
• 3D visual grounding

Innovation

• equation-anchored tool-use
• camera-robust 3D localization
• pinhole back-projection
• multimodal LLMs
• 3D visual grounding