Are We Ready for RL in Text-to-3D Generation? A Progressive Investigation

2025-12-11

AI Summary

This work addresses three key challenges in text-to-3D generation: poor geometric consistency, low texture fidelity, and difficulty in optimizing global structure. To this end, it pioneers the first systematic application of reinforcement learning (RL) across the entire autoregressive 3D generation pipeline. The authors propose a multidimensional RL research framework, introduce MME-3DR—the first benchmark tailored for evaluating implicit 3D reasoning—and design Hi-GRPO, a hierarchical RL algorithm integrating multimodal reward modeling, layered reward aggregation, and human preference alignment. Furthermore, they develop AR3D-R1, the first open-source, RL-enhanced text-to-3D model, which jointly optimizes autoregressive 3D representations and differentiable rendering. Experiments demonstrate that AR3D-R1 achieves a 23.6% improvement over baseline methods on MME-3DR, significantly enhancing both coarse-grained shape consistency and fine-grained texture quality.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Reinforcement learning (RL), earlier proven to be effective in large language and multi-modal models, has been successfully extended to enhance 2D image generation recently. However, applying RL to 3D generation remains largely unexplored due to the higher spatial complexity of 3D objects, which require globally consistent geometry and fine-grained local textures. This makes 3D generation significantly sensitive to reward designs and RL algorithms. To address these challenges, we conduct the first systematic study of RL for text-to-3D autoregressive generation across several dimensions. (1) Reward designs: We evaluate reward dimensions and model choices, showing that alignment with human preference is crucial, and that general multi-modal models provide robust signal for 3D attributes. (2) RL algorithms: We study GRPO variants, highlighting the effectiveness of token-level optimization, and further investigate the scaling of training data and iterations. (3) Text-to-3D Benchmarks: Since existing benchmarks fail to measure implicit reasoning abilities in 3D generation models, we introduce MME-3DR. (4) Advanced RL paradigms: Motivated by the natural hierarchy of 3D generation, we propose Hi-GRPO, which optimizes the global-to-local hierarchical 3D generation through dedicated reward ensembles. Based on these insights, we develop AR3D-R1, the first RL-enhanced text-to-3D model, expert from coarse shape to texture refinement. We hope this study provides insights into RL-driven reasoning for 3D generation. Code is released at https://github.com/Ivan-Tang-3D/3DGen-R1.

Problem

• Investigates RL application in text-to-3D generation due to spatial complexity challenges.
• Addresses reward design and algorithm sensitivity for consistent geometry and textures.
• Proposes hierarchical RL and benchmarks to enhance 3D generation reasoning.

Innovation

• Using RL for text-to-3D generation with token-level optimization
• Introducing Hi-GRPO for hierarchical global-to-local 3D generation
• Developing AR3D-R1 as first RL-enhanced text-to-3D model