CoherenDream: Boosting Holistic Text Coherence in 3D Generation via Multimodal Large Language Models Feedback

2025-04-28

AI Summary

To address the weak semantic interaction among multiple objects and degraded text-3D alignment in text-to-3D generation—caused by accumulated view-agnostic bias in Score Distillation Sampling (SDS)—this paper proposes Text-Consistent Score Distillation (TCSD), a novel optimization objective. We pioneer the integration of multimodal large language models (MLLMs) into the SDS optimization loop: (i) we design 3DLLaVA-CRITIC, a fine-grained cross-modal evaluator that provides alignment feedback between text and rendered views; and (ii) we introduce LLM-layout semantic initialization to inject robust spatial-structural priors. Evaluated on T³Bench and the TIFA subset, TCSD achieves state-of-the-art performance, significantly improving text fidelity and semantic interaction consistency in multi-object scenes, while accelerating convergence by 40%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Score Distillation Sampling (SDS) has achieved remarkable success in text-to-3D content generation. However, SDS-based methods struggle to maintain semantic fidelity for user prompts, particularly when involving multiple objects with intricate interactions. While existing approaches often address 3D consistency through multiview diffusion model fine-tuning on 3D datasets, this strategy inadvertently exacerbates text-3D alignment degradation. The limitation stems from SDS's inherent accumulation of view-independent biases during optimization, which progressively diverges from the ideal text alignment direction. To alleviate this limitation, we propose a novel SDS objective, dubbed as Textual Coherent Score Distillation (TCSD), which integrates alignment feedback from multimodal large language models (MLLMs). Our TCSD leverages cross-modal understanding capabilities of MLLMs to assess and guide the text-3D correspondence during the optimization. We further develop 3DLLaVA-CRITIC - a fine-tuned MLLM specialized for evaluating multiview text alignment in 3D generations. Additionally, we introduce an LLM-layout initialization that significantly accelerates optimization convergence through semantic-aware spatial configuration. Comprehensive evaluations demonstrate that our framework, CoherenDream, establishes state-of-the-art performance in text-aligned 3D generation across multiple benchmarks, including T$^3$Bench and TIFA subset. Qualitative results showcase the superior performance of CoherenDream in preserving textual consistency and semantic interactions. As the first study to incorporate MLLMs into SDS optimization, we also conduct extensive ablation studies to explore optimal MLLM adaptations for 3D generation tasks.

Problem

• Enhancing text-to-3D semantic fidelity for complex multi-object interactions
• Reducing text-3D alignment degradation in view-independent SDS optimization
• Improving coherence via MLLM feedback for cross-modal text-3D correspondence

Innovation

• Introduces Textual Coherent Score Distillation (TCSD) for alignment
• Develops 3DLLaVA-CRITIC for evaluating multiview text alignment
• Uses LLM-layout initialization for faster optimization convergence