Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting

2025-04-09

AI Summary

This study addresses the challenge of high-throughput phenotyping of wheat spikes in field conditions, specifically tackling accurate 3D reconstruction and instance segmentation of individual spikes under severe occlusion. We propose the first integration of 3D Gaussian Splatting (3DGS) into crop phenotyping, combined with the Segment Anything Model (SAM) to achieve end-to-end 3D instance segmentation—overcoming the accuracy and efficiency limitations of Neural Radiance Fields (NeRF) and traditional multi-view stereo (MVS) methods for modeling fine-scale plant organs within complex canopies. Leveraging multi-view RGB images, our method reconstructs dense 3D point clouds, validated against ground-truth LiDAR scans. It achieves mean absolute percentage errors of 15.1%, 18.3%, and 40.2% for spike length, width, and volume estimation, respectively—significantly outperforming state-of-the-art approaches. The framework enables non-destructive, scalable extraction of spike morphological traits, advancing high-throughput phenotyping for crop breeding.

Technology Category

• Computer Vision: 3D Computer Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Automated extraction of plant morphological traits is crucial for supporting crop breeding and agricultural management through high-throughput field phenotyping (HTFP). Solutions based on multi-view RGB images are attractive due to their scalability and affordability, enabling volumetric measurements that 2D approaches cannot directly capture. While advanced methods like Neural Radiance Fields (NeRFs) have shown promise, their application has been limited to counting or extracting traits from only a few plants or organs. Furthermore, accurately measuring complex structures like individual wheat heads-essential for studying crop yields-remains particularly challenging due to occlusions and the dense arrangement of crop canopies in field conditions. The recent development of 3D Gaussian Splatting (3DGS) offers a promising alternative for HTFP due to its high-quality reconstructions and explicit point-based representation. In this paper, we present Wheat3DGS, a novel approach that leverages 3DGS and the Segment Anything Model (SAM) for precise 3D instance segmentation and morphological measurement of hundreds of wheat heads automatically, representing the first application of 3DGS to HTFP. We validate the accuracy of wheat head extraction against high-resolution laser scan data, obtaining per-instance mean absolute percentage errors of 15.1%, 18.3%, and 40.2% for length, width, and volume. We provide additional comparisons to NeRF-based approaches and traditional Muti-View Stereo (MVS), demonstrating superior results. Our approach enables rapid, non-destructive measurements of key yield-related traits at scale, with significant implications for accelerating crop breeding and improving our understanding of wheat development.

Problem

• Automated 3D reconstruction of wheat heads in field conditions
• Precise instance segmentation for dense crop canopies
• High-throughput phenotyping of yield-related wheat traits

Innovation

• Uses 3D Gaussian Splatting for reconstruction
• Leverages Segment Anything Model for segmentation
• Automates phenotyping of wheat heads