Hybrid Decision Making via Conformal VLM-generated Guidance

2026-04-16

AI Summary

Existing Learning-to-Guide (LtG) approaches often produce redundant and overly complex guidance, which impairs human decision-making efficiency. This work proposes ConfGuide, a novel method that, while preserving human authority over final decisions, introduces conformal risk control into the LtG framework for the first time. ConfGuide dynamically regulates the false negative rate to selectively highlight critical findings and leverages vision-language models to generate concise, focused textual guidance. Evaluated in multi-label medical diagnosis settings, the method achieves controllable and reliable guidance generation, significantly enhancing both readability and decision-support efficacy.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Language and Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Building on recent advances in AI, hybrid decision making (HDM) holds the promise of improving human decision quality and reducing cognitive load. We work in the context of learning to guide (LtG), a recently proposed HDM framework in which the human is always responsible for the final decision: rather than suggesting decisions, in LtG the AI supplies (textual) guidance useful for facilitating decision making. One limiting factor of existing approaches is that their guidance compounds information about all possible outcomes, and as a result it can be difficult to digest. We address this issue by introducing ConfGuide, a novel LtG approach that generates more succinct and targeted guidance. To this end, it employs conformal risk control to select a set of outcomes, ensuring a cap on the false negative rate. We demonstrate our approach on a real-world multi-label medical diagnosis task. Our empirical evaluation highlights the promise of ConfGuide.

Problem

• hybrid decision making
• learning to guide
• AI-generated guidance
• conformal risk control
• medical diagnosis

Innovation

• Conformal Risk Control
• Learning to Guide
• Hybrid Decision Making
• Targeted Guidance
• False Negative Rate