Signal Fidelity Index-Aware Calibration for Dementia Predictions Across Heterogeneous Real-World Data

2025-09-10

AI Summary

Predictive models for dementia exhibit degraded generalizability across heterogeneous electronic health record (EHR) systems due to diagnostic signal attenuation—arising from inter-institutional variations in diagnostic quality and temporal consistency. Method: We propose an unsupervised calibration framework grounded in the Signal Fidelity Index (SFI), the first patient-level metric incorporating six interpretable dimensions—including diagnostic specificity and temporal consistency—enabling label-free, adaptive assessment of data reliability. Integrating epidemiology-informed synthetic data simulation with a multiplicative calibration strategy, we optimize calibration parameters via batch-wise tuning (α = 2.0). Results: On real-world heterogeneous EHR data, SFI-aware calibration improves balanced accuracy, recall, F1-score, and detection rate by 10.3%, 32.5%, 26.1%, and 41.1%, respectively—approaching supervised baseline performance. This establishes a novel paradigm for label-efficient, cross-institutional transfer learning in clinical predictive modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

extbf{Background:} Machine learning models trained on electronic health records (EHRs) often degrade across healthcare systems due to distributional shift. A fundamental but underexplored factor is diagnostic signal decay: variability in diagnostic quality and consistency across institutions, which affects the reliability of codes used for training and prediction. extbf{Objective:} To develop a Signal Fidelity Index (SFI) quantifying diagnostic data quality at the patient level in dementia, and to test SFI-aware calibration for improving model performance across heterogeneous datasets without outcome labels. extbf{Methods:} We built a simulation framework generating 2,500 synthetic datasets, each with 1,000 patients and realistic demographics, encounters, and coding patterns based on dementia risk factors. The SFI was derived from six interpretable components: diagnostic specificity, temporal consistency, entropy, contextual concordance, medication alignment, and trajectory stability. SFI-aware calibration applied a multiplicative adjustment, optimized across 50 simulation batches. extbf{Results:} At the optimal parameter ($α$ = 2.0), SFI-aware calibration significantly improved all metrics (p $<$ 0.001). Gains ranged from 10.3% for Balanced Accuracy to 32.5% for Recall, with notable increases in Precision (31.9%) and F1-score (26.1%). Performance approached reference standards, with F1-score and Recall within 1% and Balanced Accuracy and Detection Rate improved by 52.3% and 41.1%, respectively. extbf{Conclusions:} Diagnostic signal decay is a tractable barrier to model generalization. SFI-aware calibration provides a practical, label-free strategy to enhance prediction across healthcare contexts, particularly for large-scale administrative datasets lacking outcome labels.

Problem

• Addressing diagnostic signal decay in EHR data across institutions
• Quantifying patient-level diagnostic data quality for dementia predictions
• Improving model generalization without outcome labels via calibration

Innovation

• Developed Signal Fidelity Index for data quality
• Applied multiplicative calibration using SFI adjustment
• Used simulation framework with synthetic datasets