Score Distributions, Not Cells: Evaluating Single-Cell Perturbations Under Class Overlap

2026-07-05

AI Summary

This work addresses the challenge that in single-cell perturbation data, cell populations under different perturbations exhibit substantial overlap, rendering conventional single-cell classification accuracy an unreliable metric of model performance. To overcome this limitation, the authors propose the Classifier Discrimination Score (CDS), which constructs a perturbation-level profile by aggregating classifier output probability distributions across entire cell populations and replaces single-cell predictions with population-level ranking. Remarkably, CDS recovers near-perfect perturbation identification from weak classifiers without requiring retraining. The method is compatible with diverse architectures—including linear models, MLPs, and Transformers—and demonstrates significant gains in identification accuracy on the Tahoe-100M and Virtual Cell Challenge datasets, with particularly pronounced advantages in low-cell-count regimes.

Abstract

Most classification problems assume the classes are roughly separable, so that an individual sample can usually be assigned to one class. Single-cell perturbation data violates this assumption: two perturbations can produce different populations of cells while overlapping so much that an individual cell could belong to either. Per-cell accuracy then measures this overlap rather than model quality. We see this on Tahoe-100M and the Virtual Cell Challenge, where a linear classifier, an MLP, and a Transformer all plateau near macro-F1 0.2-0.3 even though almost every pair of perturbations is statistically distinguishable. The fix is to score perturbations across the whole population rather than cell by cell. We average a classifier's per-cell probability vectors over all cells of a perturbation to form a population profile, then rank candidate perturbations by this profile; we call the resulting score the Classifier Discrimination Score (CDS). Taking the top-ranked class recovers the winning perturbation. It needs no retraining, costs linear time in the number of cells, and recovers near-perfect identification from the same weak models. CDS differs from the pseudobulk-based Perturbation Discrimination Score (PDS) used in recent benchmarks only in where the average is taken, raw gene expression for PDS versus a learned discriminative space for CDS, and identifies the true perturbation more reliably on both datasets, with the gap widening as cells grow scarce. Because a metric that misranks the ground truth will misrank the models scored against it, per-cell accuracy and raw-pseudobulk scores should be used with caution when comparing perturbation models.

Problem

• single-cell perturbations
• class overlap
• score distributions
• model evaluation
• classification accuracy

Innovation

• Classifier Discrimination Score
• single-cell perturbation
• population-level evaluation
• class overlap
• pseudobulk