Coherent Hierarchical Multi-Label Learning to Defer for Medical Imaging

2026-05-04

AI Summary

Existing Learn-to-Defer (L2D) approaches in medical image analysis neglect the hierarchical structure of labels, leading to classification inconsistencies and logical contradictions in deferral decisions. This work formalizes, for the first time, the problem of hierarchically consistent deferral, modeling deferral as a semantically coherent delegation act and deriving a Bayes-optimal consistent deferral rule that reveals even node-level L2D can remain inconsistent. To address this, we propose TBP+RPO, a joint action model featuring exact consistency projection and contract-aware learning: exact projection via dynamic programming fully eliminates inconsistencies, while Taxonomy-aware Belief Propagation (TBP) and Recursive Policy Optimization (RPO) enable end-to-end training. Experiments demonstrate that TBP+RPO nearly eradicates inconsistency on both real-world and simulated medical imaging benchmarks while maintaining superior performance.

Abstract

Learning to Defer (L2D) enables a model to predict autonomously or defer to an expert, but prior work largely assumes flat label spaces. We study the first L2D setting with hierarchical multi-label decisions, motivated by medical-imaging workflows in which findings are organised by clinical taxonomies. In this setting, deferral is a delegation action rather than a label assignment, so treating it as an independent per-label decision can produce deferral incoherence, including taxonomic contradictions, delegation violations, and deferrals of labels already implied by the model's own assertions. We formalise coherent hierarchical deferral under a Selective-Exclusion handoff contract, characterise the Bayes-optimal coherent deferral rule, and show that even nodewise Bayes L2D can be action-incoherent. We then propose two remedies: exact coherent projection, a dynamic-programming decoder over the coherent action set, and Taxonomic Belief Propagation (TBP) with Recursive Policy Optimisation (RPO), a contract-aware joint action model trained through the same recursion used at inference. Across real-reader and controlled-expert medical-imaging benchmarks, naive binary-relevance L2D exhibits non-trivial incoherence. Projection removes it exactly, and fast TBP+RPO drives incoherence near zero while retaining strong utility.

Problem

• Learning to Defer
• Hierarchical Multi-Label Classification
• Medical Imaging
• Deferral Coherence
• Taxonomic Contradictions

Innovation

• Learning to Defer
• Hierarchical Multi-Label Classification
• Coherent Deferral
• Taxonomic Belief Propagation
• Selective-Exclusion Contract