PlantDreamer: Achieving Realistic 3D Plant Models with Diffusion-Guided Gaussian Splatting

2025-05-21

AI Summary

Current 3D generative models struggle to capture the intricate geometry and fine-scale textures of plants, limiting their utility in plant phenotyping. To address this, we propose the first text-driven diffusion-based 3D generation framework integrating Gaussian Splatting. Our method introduces four key innovations: (1) an L-system-derived mesh prior encoding botanical structural regularity; (2) depth-conditioned ControlNet for precise geometric guidance; (3) LoRA-based efficient fine-tuning for domain adaptation; and (4) an adaptive Gaussian pruning algorithm for optimized splat distribution. The framework supports both text-to-3D plant synthesis and end-to-end Gaussian enhancement of real-world point clouds. Experiments demonstrate substantial improvements over state-of-the-art text-to-3D methods in geometric fidelity, texture realism, and rendering quality. Furthermore, our approach successfully upgrades existing agricultural point cloud datasets, enabling high-precision 3D phenotypic analysis.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Recent years have seen substantial improvements in the ability to generate synthetic 3D objects using AI. However, generating complex 3D objects, such as plants, remains a considerable challenge. Current generative 3D models struggle with plant generation compared to general objects, limiting their usability in plant analysis tools, which require fine detail and accurate geometry. We introduce PlantDreamer, a novel approach to 3D synthetic plant generation, which can achieve greater levels of realism for complex plant geometry and textures than available text-to-3D models. To achieve this, our new generation pipeline leverages a depth ControlNet, fine-tuned Low-Rank Adaptation and an adaptable Gaussian culling algorithm, which directly improve textural realism and geometric integrity of generated 3D plant models. Additionally, PlantDreamer enables both purely synthetic plant generation, by leveraging L-System-generated meshes, and the enhancement of real-world plant point clouds by converting them into 3D Gaussian Splats. We evaluate our approach by comparing its outputs with state-of-the-art text-to-3D models, demonstrating that PlantDreamer outperforms existing methods in producing high-fidelity synthetic plants. Our results indicate that our approach not only advances synthetic plant generation, but also facilitates the upgrading of legacy point cloud datasets, making it a valuable tool for 3D phenotyping applications.

Problem

• Generating realistic 3D plant models with AI
• Improving textural realism and geometric integrity
• Enhancing legacy plant point cloud datasets

Innovation

• Uses diffusion-guided Gaussian splatting for realism
• Integrates depth ControlNet and Low-Rank Adaptation
• Enhances real-world plant point clouds conversion