ScaleMAI: Accelerating the Development of Trusted Datasets and AI Models

2025-01-06

AI Summary

Medical AI development is hindered by lengthy dataset curation cycles and the decoupling of annotation from model training. To address this, we propose an AI-driven collaborative co-evolution framework for medical data, instantiated on pancreatic tumor CT analysis. Our approach introduces a novel human-in-the-loop, progressive “data flywheel” mechanism that jointly enhances annotation quality and model performance. Methodologically, it integrates multi-round human-in-the-loop iteration, 3D voxel-level semi-automatic annotation, domain-adaptive few-shot learning, and cross-task joint modeling (detection, segmentation, classification). We construct a high-quality, multi-task dataset comprising 25,362 CT scans. Our flagship model achieves annotation accuracy comparable to that of experts with 30 years of experience, delivering performance gains of 14%, 5%, and 72% over prior state-of-the-art on detection, segmentation, and classification benchmarks, respectively. This work transcends static dataset paradigms, enabling dynamic, scalable, and trustworthy medical AI infrastructure.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Building trusted datasets is critical for transparent and responsible Medical AI (MAI) research, but creating even small, high-quality datasets can take years of effort from multidisciplinary teams. This process often delays AI benefits, as human-centric data creation and AI-centric model development are treated as separate, sequential steps. To overcome this, we propose ScaleMAI, an agent of AI-integrated data curation and annotation, allowing data quality and AI performance to improve in a self-reinforcing cycle and reducing development time from years to months. We adopt pancreatic tumor detection as an example. First, ScaleMAI progressively creates a dataset of 25,362 CT scans, including per-voxel annotations for benign/malignant tumors and 24 anatomical structures. Second, through progressive human-in-the-loop iterations, ScaleMAI provides Flagship AI Model that can approach the proficiency of expert annotators (30-year experience) in detecting pancreatic tumors. Flagship Model significantly outperforms models developed from smaller, fixed-quality datasets, with substantial gains in tumor detection (+14%), segmentation (+5%), and classification (72%) on three prestigious benchmarks. In summary, ScaleMAI transforms the speed, scale, and reliability of medical dataset creation, paving the way for a variety of impactful, data-driven applications.

Problem

• Medical AI
• Dataset Creation
• Time-consuming Process

Innovation

• ScaleMAI
• Medical Data Curation
• AI Model Enhancement