An AI Implementation Science Study to Improve Trustworthy Data in a Large Healthcare System

2025-12-01

AI Summary

Data untrustworthiness in large-scale healthcare systems impedes clinical deployment of AI. Method: This study pilots a trustworthy AI implementation framework at Shriners Children’s, migrating its research data warehouse to OMOP CDM v5.4 and FHIR standards; building a Python-based data quality assessment tool on Microsoft Fabric; deeply integrating the METRIC trustworthy AI framework into multi-center clinical data quality evaluation; and extending OHDsi’s Data Quality Dashboard (DQD) for Python compatibility. A novel “systematic standardization + scenario-specific customization” hybrid AI implementation strategy is proposed. Contribution/Results: The framework enables cross-center data standardization and real-time quality monitoring, significantly improving data completeness, timeliness, and distributional consistency. It delivers a reproducible, governable pathway for deploying trustworthy AI in clinical use cases—exemplified by craniofacial microanomaly detection.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Humans and AI: AI for Accessibility

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The rapid growth of Artificial Intelligence (AI) in healthcare has sparked interest in Trustworthy AI and AI Implementation Science, both of which are essential for accelerating clinical adoption. However, strict regulations, gaps between research and clinical settings, and challenges in evaluating AI systems continue to hinder real-world implementation. This study presents an AI implementation case study within Shriners Childrens (SC), a large multisite pediatric system, showcasing the modernization of SCs Research Data Warehouse (RDW) to OMOP CDM v5.4 within a secure Microsoft Fabric environment. We introduce a Python-based data quality assessment tool compatible with SCs infrastructure, extending OHDsi's R/Java-based Data Quality Dashboard (DQD) and integrating Trustworthy AI principles using the METRIC framework. This extension enhances data quality evaluation by addressing informative missingness, redundancy, timeliness, and distributional consistency. We also compare systematic and case-specific AI implementation strategies for Craniofacial Microsomia (CFM) using the FHIR standard. Our contributions include a real-world evaluation of AI implementations, integration of Trustworthy AI principles into data quality assessment, and insights into hybrid implementation strategies that blend systematic infrastructure with use-case-driven approaches to advance AI in healthcare.

Problem

• Improving data quality in healthcare AI systems
• Integrating Trustworthy AI principles into data assessment
• Comparing systematic and case-specific AI implementation strategies

Innovation

• Modernizing data warehouse to OMOP CDM in secure cloud environment
• Extending data quality tool with Python for Trustworthy AI principles
• Comparing systematic and case-specific AI implementation strategies using FHIR