From Field to Drone: Domain Drift Tolerant Automated Multi-Species and Damage Plant Semantic Segmentation for Herbicide Trials

2025-08-10

AI Summary

Traditional field trials of herbicides rely on manual visual assessment, which is labor-intensive and highly subjective. To address the challenge of automated semantic segmentation for multi-species plants and herbicide-induced phytotoxicity, this work proposes a decoupled framework integrating a general-purpose self-supervised vision model with taxonomic hierarchical reasoning—effectively mitigating domain shifts across imaging platforms (ground cameras → UAVs) and geographical locations. The method unifies self-supervised pretraining, end-to-end semantic segmentation, and hierarchical classification inference, trained and validated on multi-source, multi-temporal imagery. It achieves an F1-score of 0.85 (R² = 0.98) for plant species identification and 0.44 (R² = 0.87) for phytotoxicity classification. Crucially, it demonstrates robust cross-domain generalization on UAV data and has been deployed in BASF’s large-scale phenotyping platform—the first engineering implementation of cross-platform semantic segmentation for agricultural phenotyping.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Field trials are vital in herbicide research and development to assess effects on crops and weeds under varied conditions. Traditionally, evaluations rely on manual visual assessments, which are time-consuming, labor-intensive, and subjective. Automating species and damage identification is challenging due to subtle visual differences, but it can greatly enhance efficiency and consistency. We present an improved segmentation model combining a general-purpose self-supervised visual model with hierarchical inference based on botanical taxonomy. Trained on a multi-year dataset (2018-2020) from Germany and Spain using digital and mobile cameras, the model was tested on digital camera data (year 2023) and drone imagery from the United States, Germany, and Spain (year 2024) to evaluate robustness under domain shift. This cross-device evaluation marks a key step in assessing generalization across platforms of the model. Our model significantly improved species identification (F1-score: 0.52 to 0.85, R-squared: 0.75 to 0.98) and damage classification (F1-score: 0.28 to 0.44, R-squared: 0.71 to 0.87) over prior methods. Under domain shift (drone images), it maintained strong performance with moderate degradation (species: F1-score 0.60, R-squared 0.80; damage: F1-score 0.41, R-squared 0.62), where earlier models failed. These results confirm the model's robustness and real-world applicability. It is now deployed in BASF's phenotyping pipeline, enabling large-scale, automated crop and weed monitoring across diverse geographies.

Problem

• Automate plant species and damage segmentation for herbicide trials
• Overcome domain drift in multi-year, multi-location datasets
• Improve robustness across digital, mobile, and drone imagery

Innovation

• Self-supervised visual model for segmentation
• Hierarchical inference based on taxonomy
• Cross-device domain shift robustness