RareAlert: Aligning heterogeneous large language model reasoning for early rare disease risk screening

2026-01-26

AI Summary

This work proposes a novel approach to address the challenges of missed or delayed diagnoses in the initial evaluation of rare diseases, where limited clinical information and high uncertainty often impede accurate assessment. By leveraging routinely available preliminary data, the method integrates reasoning pathways from ten leading large language models—including GPT-5 and Claude—through machine learning–based calibration and weighting, and distills this ensemble into a lightweight, locally deployable model fine-tuned from Qwen3-4B. This enables population-wide early screening for rare disease risk. Notably, it is the first application of multi-source, heterogeneous large-model reasoning alignment to mitigate clinical diagnostic uncertainty. Evaluated on an independent test set, the approach achieves an AUC of 0.917, significantly outperforming both individual large models and conventional ensemble methods, thereby demonstrating its effectiveness and innovation in high-uncertainty diagnostic settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Missed and delayed diagnosis remains a major challenge in rare disease care. At the initial clinical encounters, physicians assess rare disease risk using only limited information under high uncertainty. When high-risk patients are not recognised at this stage, targeted diagnostic testing is often not initiated, resulting in missed diagnosis. Existing primary care triage processes are structurally insufficient to reliably identify patients with rare diseases at initial clinical presentation and universal screening is needed to reduce diagnostic delay. Here we present RareAlert, an early screening system which predict patient-level rare disease risk from routinely available primary-visit information. RareAlert integrates reasoning generated by ten LLMs, calibrates and weights these signals using machine learning, and distils the aligned reasoning into a single locally deployable model. To develop and evaluate RareAlert, we curated RareBench, a real-world dataset of 158,666 cases covering 33 Orphanet disease categories and more than 7,000 rare conditions, including both rare and non-rare presentations. The results showed that rare disease identification can be reconceptualised as a universal uncertainty resolution process applied to the general patient population. On an independent test set, RareAlert, a Qwen3-4B based model trained with calibrated reasoning signals, achieved an AUC of 0.917, outperforming the best machine learning ensemble and all evaluated LLMs, including GPT-5, DeepSeek-R1, Claude-3.7-Sonnet, o3-mini, Gemini-2.5-Pro, and Qwen3-235B. These findings demonstrate the diversity in LLM medical reasoning and the effectiveness of aligning such reasoning in highly uncertain clinical tasks. By incorporating calibrated reasoning into a single model, RareAlert enables accurate, privacy-preserving, and scalable rare disease risk screening suitable for large-scale local deployment.

Problem

• rare disease
• diagnostic delay
• early screening
• clinical uncertainty
• missed diagnosis

Innovation

• reasoning alignment
• heterogeneous LLMs
• rare disease screening
• calibrated reasoning
• knowledge distillation