Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation

2025-09-29

AI Summary

To address low labeling efficiency for soil profile data under limited expert annotation resources, this paper proposes an uncertainty-guided human-in-the-loop annotation framework. Methodologically, we first integrate model-agnostic conformal prediction into SoilNet—a multimodal, multitask deep regression model for soil depth estimation—to achieve reliable and well-calibrated uncertainty quantification. Based on these uncertainty estimates, we design a budget-constrained active annotation pipeline that dynamically triggers expert intervention only when model prediction confidence falls below a predefined threshold. Our contributions are: (1) the first conformalized uncertainty calibration scheme tailored to soil profile regression; and (2) statistically significant improvement in regression accuracy (p < 0.01) under identical annotation budgets, while maintaining classification performance comparable to baselines—demonstrating the effectiveness of uncertainty-driven optimization of expert labeling effort.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Uncertainty quantification is essential in human-machine collaboration, as human agents tend to adjust their decisions based on the confidence of the machine counterpart. Reliably calibrated model uncertainties, hence, enable more effective collaboration, targeted expert intervention and more responsible usage of Machine Learning (ML) systems. Conformal prediction has become a well established model-agnostic framework for uncertainty calibration of ML models, offering statistically valid confidence estimates for both regression and classification tasks. In this work, we apply conformal prediction to $ extit{SoilNet}$, a multimodal multitask model for describing soil profiles. We design a simulated human-in-the-loop (HIL) annotation pipeline, where a limited budget for obtaining ground truth annotations from domain experts is available when model uncertainty is high. Our experiments show that conformalizing SoilNet leads to more efficient annotation in regression tasks and comparable performance scores in classification tasks under the same annotation budget when tested against its non-conformal counterpart. All code and experiments can be found in our repository: https://github.com/calgo-lab/BGR

Problem

• Develops uncertainty-guided human-AI collaboration for soil annotation
• Applies conformal prediction to calibrate SoilNet model uncertainties
• Enables efficient soil horizon annotation with limited expert budget

Innovation

• Applied conformal prediction for uncertainty calibration
• Designed human-in-the-loop annotation pipeline
• Used uncertainty to guide expert intervention