When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models

2026-03-22

AI Summary

This study addresses the significant heterogeneity in patient education materials across U.S. transplant centers and the absence of scalable quantitative assessment methods. The authors propose the first systematic framework that integrates a retrieval-augmented language model with a five-label consistency taxonomy to enable multidimensional alignment analysis across questions, topics, organ types, and centers. Applied to 102 patient handbooks and 1,115 patient-generated questions, the approach reveals clinically significant discrepancies in 20.8% of cases and identifies content gaps in 96.2% of question–handbook pairs—most notably, a 95.1% omission rate for reproductive health information. The framework further establishes the first interpretable and stable center-level discrepancy profiles, providing a data-driven foundation to improve consistency in transplant education.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Evaluation and Analysis
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Patient education materials for solid-organ transplantation vary substantially across U.S. centers, yet no systematic method exists to quantify this heterogeneity at scale. We introduce a framework that grounds the same patient questions in different centers' handbooks using retrieval-augmented language models and compares the resulting answers using a five-label consistency taxonomy. Applied to 102 handbooks from 23 centers and 1,115 benchmark questions, the framework quantifies heterogeneity across four dimensions: question, topic, organ, and center. We find that 20.8% of non-absent pairwise comparisons exhibit clinically meaningful divergence, concentrated in condition monitoring and lifestyle topics. Coverage gaps are even more prominent: 96.2% of question-handbook pairs miss relevant content, with reproductive health at 95.1% absence. Center-level divergence profiles are stable and interpretable, where heterogeneity reflects systematic institutional differences, likely due to patient diversity. These findings expose an information gap in transplant patient education materials, with document-grounded medical question answering highlighting opportunities for content improvement.

Problem

• institutional variation
• transplant guidance
• patient education materials
• information heterogeneity
• clinical divergence

Innovation

• retrieval-augmented language models
• institutional variation
• transplant patient education
• document-grounded QA
• consistency taxonomy