PhenoBench: A Comprehensive Benchmark for Cell Phenotyping

2025-07-04

AI Summary

In digital pathology, existing foundation models lack a unified, clinically representative benchmark for evaluating generalization in cellular phenotyping. To address this, we introduce PhenoBench—the first fine-grained cellular phenotyping benchmark for H&E-stained whole-slide images—comprising PhenoCell, a high-quality, multi-omics–validated dataset with annotations for 14 cell types, and an open-source, standardized framework for fine-tuning and evaluation. Systematic evaluation reveals that state-of-the-art models achieve F1 scores >0.70 on conventional benchmarks (e.g., Lizard, PanNuke) but drop sharply to ~0.20 on PhenoCell, exposing a critical clinical generalization gap. By incorporating more realistic data distributions and demanding dense pixel-level prediction tasks, PhenoBench establishes a significantly more challenging and clinically relevant evaluation standard. This benchmark advances the alignment of pathology foundation models with real-world clinical complexity.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Other Foundations of Computer Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Digital pathology has seen the advent of a wealth of foundational models (FM), yet to date their performance on cell phenotyping has not been benchmarked in a unified manner. We therefore propose PhenoBench: A comprehensive benchmark for cell phenotyping on Hematoxylin and Eosin (H&E) stained histopathology images. We provide both PhenoCell, a new H&E dataset featuring 14 granular cell types identified by using multiplexed imaging, and ready-to-use fine-tuning and benchmarking code that allows the systematic evaluation of multiple prominent pathology FMs in terms of dense cell phenotype predictions in different generalization scenarios. We perform extensive benchmarking of existing FMs, providing insights into their generalization behavior under technical vs. medical domain shifts. Furthermore, while FMs achieve macro F1 scores > 0.70 on previously established benchmarks such as Lizard and PanNuke, on PhenoCell, we observe scores as low as 0.20. This indicates a much more challenging task not captured by previous benchmarks, establishing PhenoCell as a prime asset for future benchmarking of FMs and supervised models alike. Code and data are available on GitHub.

Problem

• Benchmarking cell phenotyping performance of foundational models
• Evaluating model generalization under domain shifts
• Providing a challenging dataset for future FM assessments

Innovation

• New H&E dataset with 14 cell types
• Ready-to-use fine-tuning and benchmarking code
• Evaluates FMs under domain shifts