A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics

2024-07-18

arXiv.org

AI Summary

This work addresses the challenge of incomplete 3D fruit shape reconstruction in complex greenhouse environments due to severe occlusion, hindering agricultural robotics. To this end, we introduce the first agriculture-specific benchmark for fruit 3D shape completion. Methodologically, we construct a large-scale RGB-D dataset comprising nearly 7,000 frames, paired with high-fidelity ground-truth point clouds acquired via laser scanning, covering both laboratory and real-world greenhouse settings. We develop robust pipelines for camera–LiDAR calibration, color point cloud registration, and semantic segmentation annotation, and deploy an online evaluation platform featuring a hidden test set. Our contributions are threefold: (1) the first publicly available, agriculture-oriented 3D fruit completion dataset, containing >100 individual bell peppers; (2) an open, reproducible high-precision 3D measurement framework; and (3) enabling geometric reconstruction and quantitative evaluation of multiple models under realistic occlusion conditions.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As the world population is expected to reach 10 billion by 2050, our agricultural production system needs to double its productivity despite a decline of human workforce in the agricultural sector. Autonomous robotic systems are one promising pathway to increase productivity by taking over labor-intensive manual tasks like fruit picking. To be effective, such systems need to monitor and interact with plants and fruits precisely, which is challenging due to the cluttered nature of agricultural environments causing, for example, strong occlusions. Thus, being able to estimate the complete 3D shapes of objects in presence of occlusions is crucial for automating operations such as fruit harvesting. In this paper, we propose the first publicly available 3D shape completion dataset for agricultural vision systems. We provide an RGB-D dataset for estimating the 3D shape of fruits. Specifically, our dataset contains RGB-D frames of single sweet peppers in lab conditions but also in a commercial greenhouse. For each fruit, we additionally collected high-precision point clouds that we use as ground truth. For acquiring the ground truth shape, we developed a measuring process that allows us to record data of real sweet pepper plants, both in the lab and in the greenhouse with high precision, and determine the shape of the sensed fruits. We release our dataset, consisting of almost 7,000 RGB-D frames belonging to more than 100 different fruits. We provide segmented RGB-D frames, with camera intrinsics to easily obtain colored point clouds, together with the corresponding high-precision, occlusion-free point clouds obtained with a high-precision laser scanner. We additionally enable evaluation of shape completion approaches on a hidden test set through a public challenge on a benchmark server.

Problem

• Addresses 3D shape completion for fruits in agriculture.
• Focuses on overcoming occlusions in cluttered environments.
• Provides a dataset for improving robotic fruit harvesting.

Innovation

• First public 3D shape completion dataset
• RGB-D dataset for fruit shape estimation
• High-precision laser scanner for ground truth