Topic-aware Large Language Models for Summarizing the Lived Healthcare Experiences Described in Health Stories
🤖 AI Summary
This study addresses the challenge of systematically identifying determinants of health outcomes and intervention pathways from unstructured clinical narratives authored by African American (AA) patients. We propose a topic-aware hierarchical summarization framework: first applying Latent Dirichlet Allocation (LDA) to extract cross-narrative thematic structures, then leveraging open-source large language models (LLMs) to generate interpretable, theme-level summaries, and finally employing GPT-4 to automatically assess summary fidelity and comprehensiveness. Evaluated on 50 real-world patient narratives, our method identifies 26 semantically coherent, clinically meaningful themes. Generated summaries exhibit no hallucination, high factual accuracy, and thematic completeness. Inter-rater agreement between GPT-4 evaluations and domain expert judgments is moderate to substantial (Cohen’s κ = 0.62–0.81). To our knowledge, this is the first approach enabling scalable, interpretable, and theme-driven automation of “lived experience” extraction from patient narratives—establishing a novel computational paradigm and empirical foundation for health disparities research.
Application Category
📝 Abstract
Storytelling is a powerful form of communication and may provide insights into factors contributing to gaps in healthcare outcomes. To determine whether Large Language Models (LLMs) can identify potential underlying factors and avenues for intervention, we performed topic-aware hierarchical summarization of narratives from African American (AA) storytellers. Fifty transcribed stories of AA experiences were used to identify topics in their experience using the Latent Dirichlet Allocation (LDA) technique. Stories about a given topic were summarized using an open-source LLM-based hierarchical summarization approach. Topic summaries were generated by summarizing across story summaries for each story that addressed a given topic. Generated topic summaries were rated for fabrication, accuracy, comprehensiveness, and usefulness by the GPT4 model, and the model's reliability was validated against the original story summaries by two domain experts. 26 topics were identified in the fifty AA stories. The GPT4 ratings suggest that topic summaries were free from fabrication, highly accurate, comprehensive, and useful. The reliability of GPT ratings compared to expert assessments showed moderate to high agreement. Our approach identified AA experience-relevant topics such as health behaviors, interactions with medical team members, caregiving and symptom management, among others. Such insights could help researchers identify potential factors and interventions by learning from unstructured narratives in an efficient manner-leveraging the communicative power of storytelling. The use of LDA and LLMs to identify and summarize the experience of AA individuals suggests a variety of possible avenues for health research and possible clinical improvements to support patients and caregivers, thereby ultimately improving health outcomes.
Problem
Research questions and friction points this paper is trying to address.
Identifying healthcare experience topics from African American stories using LDA
Summarizing patient narratives through hierarchical LLM approaches efficiently
Evaluating topic summaries for accuracy and usefulness in healthcare research
Innovation
Methods, ideas, or system contributions that make the work stand out.
Using LDA to identify topics in healthcare narratives
Applying hierarchical LLM summarization for story analysis
Validating summaries with GPT4 and expert assessments
🔎 Similar Papers
No similar papers found.
