Diagnostic Reasoning in Natural Language: Computational Model and Application

2024-09-09

arXiv.org

AI Summary

Existing text evaluation methods are largely black-box, lacking interpretability and human-AI collaboration capabilities. Method: This paper proposes NL-DAR, a diagnostic abductive reasoning framework for natural language tasks, which systematically integrates Pearl’s structural causal model into textual diagnostic reasoning—uniquely bridging causal modeling with linguistic understanding to enable structured, interpretable text assessment decisions. Contribution/Results: Using biomedical paper evaluation as an empirical case study, we construct the first NL-DAR benchmark dataset, uncovering expert diagnostic reasoning patterns. Through large language model (LLM) behavioral analysis, we rigorously validate NL-DAR’s efficacy and limitations in structured diagnostic tasks. We publicly release the modeling paradigm, analytical tools, and annotated resources to advance research in explainable NLP and human-AI collaborative evaluation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Diagnostic reasoning is a key component of expert work in many domains. It is a hard, time-consuming activity that requires expertise, and AI research has investigated the ways automated systems can support this process. Yet, due to the complexity of natural language, the applications of AI for diagnostic reasoning to language-related tasks are lacking. To close this gap, we investigate diagnostic abductive reasoning (DAR) in the context of language-grounded tasks (NL-DAR). We propose a novel modeling framework for NL-DAR based on Pearl's structural causal models and instantiate it in a comprehensive study of scientific paper assessment in the biomedical domain. We use the resulting dataset to investigate the human decision-making process in NL-DAR and determine the potential of LLMs to support structured decision-making over text. Our framework, open resources and tools lay the groundwork for the empirical study of collaborative diagnostic reasoning in the age of LLMs, in the scholarly domain and beyond.

Problem

• Modeling text assessment as step-wise reasoning for interpretability
• Breaking down assessment using causality theory and expert data
• Studying expert-AI collaboration in biomedical paper evaluation

Innovation

• Structured reasoning framework for text assessment
• Graph-based causality theory application
• Expert interaction data for AI collaboration