Assessing workflow impact and clinical utility of AI-assisted brain aneurysm detection: a multi-reader study

2025-03-22

AI Summary

Despite high algorithmic performance, the clinical utility of AI-assisted cerebral aneurysm detection in real-world workflows remains unclear. Method: A double-blind, multi-reader, prospective comparative study was conducted using 460 TOF-MRA scans, evaluating diagnostic sensitivity, reading time, and diagnostic confidence of both senior and junior radiologists with and without AI assistance. Contribution/Results: Although the AI model achieved state-of-the-art performance (74% sensitivity, 1.6% false positive rate), it conferred no statistically significant improvement in sensitivity for junior (p = 0.59) or senior (p = 1.0) readers. Average reading time increased by 15 seconds (p < 0.001), and diagnostic confidence remained unchanged. This study provides the first systematic evidence of a substantial performance–utility gap in clinical AI deployment, underscoring the necessity of human–AI collaborative efficacy—rather than isolated algorithmic metrics—as the primary benchmark for clinical AI validation.

Technology Category

• Humans and AI: Brain-Sensing and Analysis
• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Hardware-aware ML

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Despite the plethora of AI-based algorithms developed for anomaly detection in radiology, subsequent integration into clinical setting is rarely evaluated. In this work, we assess the applicability and utility of an AI-based model for brain aneurysm detection comparing the performance of two readers with different levels of experience (2 and 13 years). We aim to answer the following questions: 1) Do the readers improve their performance when assisted by the AI algorithm? 2) How much does the AI algorithm impact routine clinical workflow? We reuse and enlarge our open-access, Time-Of-Flight Magnetic Resonance Angiography dataset (N=460). We use 360 subjects for training/validating our algorithm and 100 as unseen test set for the reading session. Even though our model reaches state-of-the-art results on the test set (sensitivity=74%, false positive rate=1.6), we show that neither the junior nor the senior reader significantly increase their sensitivity (p=0.59, p=1, respectively). In addition, we find that reading time for both readers is significantly higher in the"AI-assisted"setting than in the"Unassisted"(+15 seconds, on average; p=3x10^(-4) junior, p=3x10^(-5) senior). The confidence reported by the readers is unchanged across the two settings, indicating that the AI assistance does not influence the certainty of the diagnosis. Our findings highlight the importance of clinical validation of AI algorithms in a clinical setting involving radiologists. This study should serve as a reminder to the community to always examine the real-word effectiveness and workflow impact of proposed algorithms.

Problem

• Evaluates AI-assisted brain aneurysm detection clinical utility
• Assesses impact of AI on radiologists' performance and workflow
• Examines real-world effectiveness of AI algorithms in radiology

Innovation

• AI-assisted brain aneurysm detection model
• Multi-reader clinical workflow impact study
• Open-access TOF-MRA dataset utilization