Decisive: Guiding User Decisions with Optimal Preference Elicitation from Unstructured Documents

2026-04-20

AI Summary

This work addresses the challenges of information overload and inaccurate preference modeling in complex decision-making scenarios involving multi-source unstructured documents. To this end, we propose an interactive decision framework that extracts objective option scores from documents and integrates Bayesian active learning with an information gain maximization strategy to adaptively pose the minimal number of pairwise comparison queries, thereby efficiently inferring users’ implicit preferences. The approach innovatively combines document-driven reasoning with a transparent preference elicitation mechanism, significantly reducing cognitive burden while preserving decision interpretability. Experimental results demonstrate that our method achieves up to a 20% improvement in decision accuracy over strong baselines across multiple domains, substantially outperforming both general-purpose large language models and existing decision support systems.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Reasoning under Uncertainty: Sequential Decision Making

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Decision-making is a cognitively intensive task that requires synthesizing relevant information from multiple unstructured sources, weighing competing factors, and incorporating subjective user preferences. Existing methods, including large language models and traditional decision-support systems, fall short: they often overwhelm users with information or fail to capture nuanced preferences accurately. We present Decisive, an interactive decision-making framework that combines document-grounded reasoning with Bayesian preference inference. Our approach grounds decisions in an objective option-scoring matrix extracted from source documents, while actively learning a user's latent preference vector through targeted elicitation. Users answer pairwise tradeoff questions adaptively selected to maximize information gain over the final decision. This process converges efficiently, minimizing user effort while ensuring recommendations remain transparent and personalized. Through extensive experiments, we demonstrate that our approach significantly outperforms both general-purpose LLMs and existing decision-making frameworks achieving up to 20% improvement in decision accuracy over strong baselines across domains.

Problem

• decision-making
• preference elicitation
• unstructured documents
• user preferences
• information overload

Innovation

• preference elicitation
• Bayesian inference
• document-grounded reasoning
• interactive decision-making
• pairwise tradeoff