Utilising Deep Learning to Elicit Expert Uncertainty

2025-01-21

AI Summary

To address the challenge of quantifying uncertainty in expert decision-making, this paper proposes a novel prior inference framework that integrates deep learning with Bayesian inversion. The method directly models expert reasoning from high-dimensional unstructured data—such as medical images and clinical text—and automatically extracts interpretable uncertainty representations. It employs CNNs, RNNs, or Transformers to encode raw decision evidence, followed by distributional fitting and Bayesian inverse problem solving to map expert behavior onto explainable probabilistic distributions. Evaluated on colorectal cancer risk assessment, the approach significantly improves both the quality and clinical interpretability of elicited prior distributions, overcoming the limitations of conventional methods reliant solely on tabular data. This work establishes a new paradigm for formalizing expert knowledge and enabling trustworthy AI-assisted diagnosis.

Technology Category

• Reasoning under Uncertainty: Uncertainty Representations
• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Recent work [ 14 ] has introduced a method for prior elicitation that utilizes records of expert decisions to infer a prior distribution. While this method provides a promising approach to eliciting expert uncertainty, it has only been demonstrated using tabular data, which may not entirely represent the information used by experts to make decisions. In this paper, we demonstrate how analysts can adopt a deep learning approach to utilize the method proposed in [14 ] with the actual information experts use. We provide an overview of deep learning models that can effectively model expert decision-making to elicit distributions that capture expert uncertainty and present an example examining the risk of colon cancer to show in detail how these models can be used.

Problem

• Deep Learning
• Expert Decision-making
• Colon Cancer Risk Assessment

Innovation

• Deep Learning
• Expert Decision-making
• Uncertainty Modeling