Doc2AHP: Inferring Structured Multi-Criteria Decision Models via Semantic Trees with LLMs

2026-01-23

AI Summary

This work addresses the limitations of large language models (LLMs) in complex decision-making tasks, where they often lack structural consistency and reliable reasoning, while traditional approaches like the Analytic Hierarchy Process (AHP) rely heavily on expert knowledge and do not scale well. To bridge this gap, the authors propose Doc2AHP, a novel framework that integrates AHP’s hierarchical logic into LLM-based reasoning for the first time. By leveraging multi-agent collaborative weight assignment and adaptive consistency optimization, Doc2AHP automatically constructs logically coherent multi-criteria decision models from unstructured documents without requiring human annotations. Experimental results demonstrate that the method significantly outperforms direct generation baselines in both logical completeness and downstream task accuracy, enabling non-expert users to efficiently build high-quality decision models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While Large Language Models (LLMs) demonstrate remarkable proficiency in semantic understanding, they often struggle to ensure structural consistency and reasoning reliability in complex decision-making tasks that demand rigorous logic. Although classical decision theories, such as the Analytic Hierarchy Process (AHP), offer systematic rational frameworks, their construction relies heavily on labor-intensive domain expertise, creating an"expert bottleneck"that hinders scalability in general scenarios. To bridge the gap between the generalization capabilities of LLMs and the rigor of decision theory, we propose Doc2AHP, a novel structured inference framework guided by AHP principles. Eliminating the need for extensive annotated data or manual intervention, our approach leverages the structural principles of AHP as constraints to direct the LLM in a constrained search within the unstructured document space, thereby enforcing the logical entailment between parent and child nodes. Furthermore, we introduce a multi-agent weighting mechanism coupled with an adaptive consistency optimization strategy to ensure the numerical consistency of weight allocation. Empirical results demonstrate that Doc2AHP not only empowers non-expert users to construct high-quality decision models from scratch but also significantly outperforms direct generative baselines in both logical completeness and downstream task accuracy.

Problem

• Large Language Models
• Analytic Hierarchy Process
• Multi-Criteria Decision Making
• Structural Consistency
• Expert Bottleneck

Innovation

• Doc2AHP
• Large Language Models
• Analytic Hierarchy Process
• structured inference
• multi-agent weighting