Horticultural Temporal Fruit Monitoring via 3D Instance Segmentation and Re-Identification using Point Clouds

2024-11-12

arXiv.org

AI Summary

To address the challenge of cross-temporal individual fruit tracking in orchards—complicated by dynamic changes such as growth, occlusion, and appearance/disappearance—this paper proposes an end-to-end, purely 3D point cloud temporal tracking method. The approach jointly performs 3D point cloud instance segmentation and sparse-convolution-based attentional matching, bypassing 2D projection distortions and enabling direct instance-level fruit segmentation and cross-frame re-identification in 3D space. Its core innovation lies in the first integration of a sparse-convolution-driven attention mechanism into a 3D instance segmentation framework, effectively modeling fruit deformation, pose variation, and partial observability. Evaluated on a real-world multi-temporal greenhouse strawberry point cloud dataset, the method significantly outperforms existing 2D- and 3D-based tracking approaches, achieving high-accuracy, robust, individual-level monitoring of fruit growth states and long-term tracking.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Tracking, profiling, and countermeasures against them
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Robotic fruit monitoring is a key step toward automated agricultural production systems. Robots can significantly enhance plant and temporal fruit monitoring by providing precise, high-throughput assessments that overcome the limitations of traditional manual methods. Fruit monitoring is a challenging task due to the significant variation in size, shape, orientation, and occlusion of fruits. Also, fruits may be harvested or newly grown between recording sessions. Most methods are 2D image-based and they lack the 3D structure, depth, and spatial information, which represent key aspects of fruit monitoring. 3D colored point clouds, instead, can offer this information but they introduce challenges such as their sparsity and irregularity. In this paper, we present a novel approach for temporal fruit monitoring that addresses point clouds collected in a greenhouse over time. Our method segments fruits using a learning-based instance segmentation approach directly on the point cloud. Each segmented fruit is processed by a 3D sparse convolutional neural network to extract descriptors, which are used in an attention-based matching network to associate fruits with their instances from previous data collections. Experimental results on a real dataset of strawberries demonstrate that our approach outperforms other methods for fruits re-identification over time, allowing for precise temporal fruit monitoring in real and complex scenarios.

Problem

• Accurate fruit monitoring in dynamic orchards over time
• Segmenting and identifying fruits in 3D colored point clouds
• Tracking fruits across different observation sessions reliably

Innovation

• 3D instance segmentation on colored point clouds
• 3D sparse CNN for discriminative fruit descriptors
• Attention-based matching network for temporal tracking