In-Field 3D Wheat Head Instance Segmentation From TLS Point Clouds Using Deep Learning Without Manual Labels

2026-03-15

AI Summary

This work addresses the challenge of wheat ear 3D instance segmentation in complex agricultural scenes, where the scarcity of manual annotations hinders supervised learning. The authors propose an unsupervised two-stage approach: first, leveraging multi-view projections combined with Grounded SAM to achieve zero-shot 2D segmentation and fuse the results into 3D pseudo-labels; second, using these pseudo-labels to train a 3D panoptic segmentation network for end-to-end wheat ear instance segmentation. This method is the first to integrate zero-shot 2D vision models with 3D deep learning without requiring any human-annotated 3D labels, effectively processing terrestrial laser scanning (TLS) point clouds. It outperforms the RGB- and 3D Gaussian Splatting-based Wheat3DGS method, demonstrating the potential of TLS for agricultural perception and showing promise for transfer to other TLS-based tasks.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

3D instance segmentation for laser scanning (LiDAR) point clouds remains a challenge in many remote sensing-related domains. Successful solutions typically rely on supervised deep learning and manual annotations, and consequently focus on objects that can be well delineated through visual inspection and manual labeling of point clouds. However, for tasks with more complex and cluttered scenes, such as in-field plant phenotyping in agriculture, such approaches are often infeasible. In this study, we tackle the task of in-field wheat head instance segmentation directly from terrestrial laser scanning (TLS) point clouds. To address the problem and circumvent the need for manual annotations, we propose a novel two-stage pipeline. To obtain the initial 3D instance proposals, the first stage uses 3D-to-2D multi-view projections, the Grounded SAM pipeline for zero-shot 2D object-centric segmentation, and multi-view label fusion. The second stage uses these initial proposals as noisy pseudo-labels to train a supervised 3D panoptic-style segmentation neural network. Our results demonstrate the feasibility of the proposed approach and show performance improvementsrelative to Wheat3DGS, a recent alternative solution for in-field wheat head instance segmentation without manual 3D annotations based on multi-view RGB images and 3D Gaussian Splatting, showcasing TLS as a competitive sensing alternative. Moreover, the results show that both stages of the proposed pipeline can deliver usable 3D instance segmentation without manual annotations, indicating promising, low-effort transferability to other comparable TLS-based point cloud segmentation tasks.

Problem

• 3D instance segmentation
• wheat head
• TLS point clouds
• manual labels
• in-field phenotyping

Innovation

• 3D instance segmentation
• terrestrial laser scanning (TLS)
• zero-shot segmentation
• pseudo-labeling
• wheat head phenotyping