RecDreamer: Consistent Text-to-3D Generation via Uniform Score Distillation

2025-02-18

AI Summary

This work addresses the “multi-face Janus” problem in text-to-3D generation—characterized by cross-view geometric inconsistency and duplicated structures—arising from score distillation’s inherent bias toward canonical poses. We propose Uniform Score Distillation (USD), a novel framework that explicitly models and corrects the marginal pose distribution to enforce uniformity, thereby eliminating canonical-pose bias. Our key innovations include: (i) a training-free pose classifier coupled with an auxiliary function to estimate and regularize pose marginals; and (ii) noise-state approximation and marginal constraint modeling to reshape pose distributions without additional training. Experiments demonstrate that USD significantly mitigates the multi-face Janus phenomenon, achieving state-of-the-art performance in multi-view consistency, geometric completeness, and text–3D alignment—outperforming existing methods across all three metrics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Current text-to-3D generation methods based on score distillation often suffer from geometric inconsistencies, leading to repeated patterns across different poses of 3D assets. This issue, known as the Multi-Face Janus problem, arises because existing methods struggle to maintain consistency across varying poses and are biased toward a canonical pose. While recent work has improved pose control and approximation, these efforts are still limited by this inherent bias, which skews the guidance during generation. To address this, we propose a solution called RecDreamer, which reshapes the underlying data distribution to achieve a more consistent pose representation. The core idea behind our method is to rectify the prior distribution, ensuring that pose variation is uniformly distributed rather than biased toward a canonical form. By modifying the prescribed distribution through an auxiliary function, we can reconstruct the density of the distribution to ensure compliance with specific marginal constraints. In particular, we ensure that the marginal distribution of poses follows a uniform distribution, thereby eliminating the biases introduced by the prior knowledge. We incorporate this rectified data distribution into existing score distillation algorithms, a process we refer to as uniform score distillation. To efficiently compute the posterior distribution required for the auxiliary function, RecDreamer introduces a training-free classifier that estimates pose categories in a plug-and-play manner. Additionally, we utilize various approximation techniques for noisy states, significantly improving system performance. Our experimental results demonstrate that RecDreamer effectively mitigates the Multi-Face Janus problem, leading to more consistent 3D asset generation across different poses.

Problem

• Addresses geometric inconsistencies in text-to-3D generation.
• Mitigates Multi-Face Janus problem in 3D asset creation.
• Ensures uniform pose distribution for consistent 3D modeling.

Innovation

• Uniform score distillation technique
• Training-free pose classifier
• Modified data distribution method