3D Hierarchical Panoptic Segmentation in Real Orchard Environments Across Different Sensors

2025-03-17

AI Summary

To address the challenge of fine-grained recognition and structural relationship modeling among individual plants, fruits, and trunks in intelligent orchard management, this paper proposes the first 3D hierarchical panoptic segmentation framework tailored for agricultural scenarios. Methodologically, it integrates Transformers and graph neural networks for multi-scale point cloud feature learning, introduces a hierarchical instance disentanglement module, and designs a structured relational modeling component, augmented by multi-sensor (TLS/RGB-D) calibration and fusion. Contributions include: (1) releasing HOPS—the first cross-sensor, real-world, multimodal annotated dataset for orchards; (2) defining and implementing three-tier outputs: semantic segmentation, fruit/trunk instance segmentation, and “single-plant” hierarchical segmentation (trunk plus associated fruits); and (3) achieving significant improvements over state-of-the-art agricultural methods in real-world deployments, enabling accurate fruit counting and plant-level yield estimation. The HOPS dataset is publicly available; code will be open-sourced shortly.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Crop yield estimation is a relevant problem in agriculture, because an accurate crop yield estimate can support farmers' decisions on harvesting or precision intervention. Robots can help to automate this process. To do so, they need to be able to perceive the surrounding environment to identify target objects. In this paper, we introduce a novel approach to address the problem of hierarchical panoptic segmentation of apple orchards on 3D data from different sensors. Our approach is able to simultaneously provide semantic segmentation, instance segmentation of trunks and fruits, and instance segmentation of plants (a single trunk with its fruits). This allows us to identify relevant information such as individual plants, fruits, and trunks, and capture the relationship among them, such as precisely estimate the number of fruits associated to each tree in an orchard. Additionally, to efficiently evaluate our approach for hierarchical panoptic segmentation, we provide a dataset designed specifically for this task. Our dataset is recorded in Bonn in a real apple orchard with a variety of sensors, spanning from a terrestrial laser scanner to a RGB-D camera mounted on different robotic platforms. The experiments show that our approach surpasses state-of-the-art approaches in 3D panoptic segmentation in the agricultural domain, while also providing full hierarchical panoptic segmentation. Our dataset has been made publicly available at https://www.ipb.uni-bonn.de/data/hops/. We will provide the open-source implementation of our approach and public competiton for hierarchical panoptic segmentation on the hidden test sets upon paper acceptance.

Problem

• Automating crop yield estimation in apple orchards using robots.
• Hierarchical panoptic segmentation of 3D data from multiple sensors.
• Identifying and relating individual plants, trunks, and fruits accurately.

Innovation

• Hierarchical panoptic segmentation for 3D orchard data
• Combines semantic and instance segmentation of plants and fruits
• Public dataset and open-source implementation provided