When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospital

2025-08-11

AI Summary

In resource-constrained settings, LLM developers lack clinical expertise while domain experts face severe time constraints, hindering effective co-design for clinical NLP applications. Method: This paper proposes a novel co-design framework for generating patient social needs summaries in safety-net hospitals. It decomposes the summarization task into semantically distinct, independently optimizable attributes and employs a multi-level cascaded LLM inference pipeline with cross-validation to align developer and clinician perspectives efficiently. Contribution/Results: The attribute-based modeling lowers the clinical knowledge barrier for developers, while cascaded validation ensures output accuracy, completeness, and traceability. Deployed in real clinical workflows, the system achieves a 3.2× improvement in social needs information extraction efficiency over baselines, satisfying both clinical utility and lightweight deployment requirements. This work establishes a reusable methodological paradigm for LLM co-design in underserved communities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Application Domains: Humanities & Computational Social Science

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have the potential to address social and behavioral determinants of health by transforming labor intensive workflows in resource-constrained settings. Creating LLM-based applications that serve the needs of underserved communities requires a deep understanding of their local context, but it is often the case that neither LLMs nor their developers possess this local expertise, and the experts in these communities often face severe time/resource constraints. This creates a disconnect: how can one engage in meaningful co-design of an LLM-based application for an under-resourced community when the communication channel between the LLM developer and domain expert is constrained? We explored this question through a real-world case study, in which our data science team sought to partner with social workers at a safety net hospital to build an LLM application that summarizes patients' social needs. Whereas prior works focus on the challenge of prompt tuning, we found that the most critical challenge in this setting is the careful and precise specification of what information to surface to providers so that the LLM application is accurate, comprehensive, and verifiable. Here we present a novel co-design framework for settings with limited access to domain experts, in which the summary generation task is first decomposed into individually-optimizable attributes and then each attribute is efficiently refined and validated through a multi-tier cascading approach.

Problem

• Bridging gap between LLM developers and time-constrained domain experts
• Ensuring LLM outputs are accurate, comprehensive, and verifiable for underserved communities
• Developing co-design framework for resource-limited settings with expert scarcity

Innovation

• Decompose summary task into optimizable attributes
• Multi-tier cascading refinement and validation
• Co-design framework for limited expert access