3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding

2026-04-09

AI Summary

This work addresses the challenge of hallucination in 3D large language model agents, which often generate responses inconsistent with the actual embodied environment—a problem inadequately mitigated by existing 2D hallucination suppression techniques. To tackle this without requiring model retraining, the authors propose a novel inference-time decoding framework that introduces visual contrastive decoding into 3D embodied intelligence. By constructing distorted 3D scene graphs through semantic substitution and geometric perturbations (e.g., coordinate or scale shifts), and leveraging object-centric structured representations, the method performs contrastive decoding between original and distorted contexts to suppress hallucinations insensitive to scene evidence. Evaluated on the 3D-POPE and HEAL benchmarks, this approach significantly enhances grounded reasoning and substantially reduces hallucination rates, all without fine-tuning the underlying model.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large multimodal models are increasingly used as the reasoning core of embodied agents operating in 3D environments, yet they remain prone to hallucinations that can produce unsafe and ungrounded decisions. Existing inference-time hallucination mitigation methods largely target 2D vision-language settings and do not transfer to embodied 3D reasoning, where failures arise from object presence, spatial layout, and geometric grounding rather than pixel-level inconsistencies. We introduce 3D-VCD, the first inference-time visual contrastive decoding framework for hallucination mitigation in 3D embodied agents. 3D-VCD constructs a distorted 3D scene graph by applying semantic and geometric perturbations to object-centric representations, such as category substitutions and coordinate or extent corruption. By contrasting predictions under the original and distorted 3D contexts, our method suppresses tokens that are insensitive to grounded scene evidence and are therefore likely driven by language priors. We evaluate 3D-VCD on the 3D-POPE and HEAL benchmarks and show that it consistently improves grounded reasoning without any retraining, establishing inference-time contrastive decoding over structured 3D representations as an effective and practical route to more reliable embodied intelligence.

Problem

• 3D hallucination
• embodied agents
• visual grounding
• 3D scene understanding
• multimodal reasoning

Innovation

• 3D-VCD
• visual contrastive decoding
• hallucination mitigation
• embodied agents
• 3D scene graph