4D Metric-Semantic Mapping for Persistent Orchard Monitoring: Method and Dataset

2024-09-29

arXiv.org

AI Summary

To address the challenge of continuous, season-spanning, single-tree and fruit-level monitoring in orchards, this paper proposes the first LiDAR-RGB-IMU fused 4D spatiotemporal metric-semantic mapping framework. Our method enables cross-seasonal 3D fruit tracking and semantic-consistent mapping via deep instance segmentation and the Hungarian algorithm, and introduces a novel 4D data association scheme supporting multimodal alignment. We release the first open-source, multimodal dataset covering the full growth cycles of five fruit tree species, with fine-grained ground-truth annotations. Evaluated in real-world orchards, our framework achieves a fruit counting error of only 3.1% (n = 1,790+), and an average fruit size estimation error of 1.1 cm—substantially advancing phenotypic analysis accuracy and horticultural resource management efficacy.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Automated persistent and fine-grained monitoring of orchards at the individual tree or fruit level helps maximize crop yield and optimize resources such as water, fertilizers, and pesticides while preventing agricultural waste. Towards this goal, we present a 4D spatio-temporal metric-semantic mapping method that fuses data from multiple sensors, including LiDAR, RGB camera, and IMU, to monitor the fruits in an orchard across their growth season. A LiDAR-RGB fusion module is designed for 3D fruit tracking and localization, which first segments fruits using a deep neural network and then tracks them using the Hungarian Assignment algorithm. Additionally, the 4D data association module aligns data from different growth stages into a common reference frame and tracks fruits spatio-temporally, providing information such as fruit counts, sizes, and positions. We demonstrate our method's accuracy in 4D metric-semantic mapping using data collected from a real orchard under natural, uncontrolled conditions with seasonal variations. We achieve a 3.1 percent error in total fruit count estimation for over 1790 fruits across 60 apple trees, along with accurate size estimation results with a mean error of 1.1 cm. The datasets, consisting of LiDAR, RGB, and IMU data of five fruit species captured across their growth seasons, along with corresponding ground truth data, will be made publicly available at: https://4d-metric-semantic-mapping.org/

Problem

• Monitoring orchards at individual tree or fruit level for phenotyping and resource optimization.
• Developing a 4D spatio-temporal mapping system to track fruit growth over time.
• Improving fruit counting accuracy and size estimation in orchard monitoring.

Innovation

• 4D spatio-temporal metric-semantic mapping system
• LiDAR-RGB fusion for 3D fruit localization
• Positional, visual, and topology-based 4D association