Benchmark Evaluation of Feredated Learning on Multi-organ Images

2026-07-09

AI Summary

This work addresses the challenges hindering the clinical deployment of medical AI—namely, stringent privacy requirements and the high heterogeneity of cross-organ, cross-modality data—by proposing MobenFL, the first large-scale, comprehensive federated learning benchmark tailored to real-world clinical settings. MobenFL integrates 20 state-of-the-art algorithms and 22 multi-center, multi-modal medical imaging datasets spanning 12 critical organs, enabling systematic evaluation across multiple dimensions including model performance, computational efficiency, and privacy preservation under complex conditions such as multi-disease scenarios and diverse imaging devices. Significantly surpassing existing benchmarks in data diversity, algorithmic breadth, and evaluation depth, MobenFL establishes a standardized framework for the reliable assessment and deployment of federated learning in medical imaging.

Abstract

The privacy requirements of medical data and its substantial variations across organs and modalities hinder the clinical implementation of medical AI. Federated learning (FL) is a feasible approach to overcome these challenges. Due to the continuous emergence of FL algorithms and the highly heterogeneous nature of medical data, objectively evaluating their performance in real-world clinical settings remains difficult. Therefore, a comprehensive federated medical imaging benchmark, serving as a unified evaluation standard, is crucial for advancing the technology toward reliable clinical application. Existing federated medical imaging benchmarks have not yet adequately incorporated state-of-the-art algorithms, are limited to data from single organs or modalities, and overly emphasize model accuracy, making it difficult to comprehensively assess the overall efficacy of FL in real-world medical environments. To address these challenges, we developed the MobenFL benchmark. This benchmark integrates 20 cutting-edge FL algorithms and 22 medical imaging datasets, covering 12 critical organs across the human body, surpassing existing benchmark in breadth. In terms of evaluation dimensions, MobenFL not only assesses performance but also systematically incorporates key metrics such as algorithmic efficiency and privacy protection capabilities. Additionally, it conducts specialized evaluations for complex real-world clinical scenarios involving different diseases, devices, and imaging modalities, thereby providing a comprehensive and in-depth evaluation framework for the clinical application of FL in the medical field.

Problem

• federated learning
• medical imaging
• benchmark evaluation
• clinical application
• data heterogeneity

Innovation

• federated learning
• medical imaging benchmark
• multi-organ
• privacy preservation
• algorithm efficiency