Effects of Distance Metrics and Scaling on the Perturbation Discrimination Score

2025-11-21

AI Summary

Perturbation discrimination score (PDS) is widely used to evaluate perturbation effects in high-dimensional gene expression data, yet its sensitivity to distance metric choice and effect-scale calibration remains poorly understood. Method: We systematically analyze PDS sensitivity through theoretical derivation and empirical evaluation, comparing ℓ₁, ℓ₂, and cosine distances—with and without feature normalization—and examine their geometric implications in high dimensions. Contribution/Results: We find that even after norm-matching, these metrics induce significant PDS discrepancies due to inherent anisotropic weighting and sparsity preferences, explainable via high-dimensional geometry. Moreover, PDS exhibits high sensitivity to effect magnitude and is vulnerable to scale-induced bias. Consequently, we propose principled design criteria for discriminative evaluation metrics: explicit joint specification of distance metric, normalization strategy, and effect-scale calibration. Our findings provide actionable theoretical foundations and practical guidelines for benchmarking and fair model evaluation in single-cell perturbation modeling.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Evaluation and Analysis
• Game Theory and Economic Paradigms: Imperfect Information

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

The Perturbation Discrimination Score (PDS) is increasingly used to evaluate whether predicted perturbation effects remain distinguishable, including in Systema and the Virtual Cell Challenge. However, its behavior in high-dimensional gene-expression settings has not been examined in detail. We show that PDS is highly sensitive to the choice of similarity or distance measure and to the scale of predicted effects. Analysis of observed perturbation responses reveals that $ell_1$ and $ell_2$-based PDS behave very differently from cosine-based measures, even after norm matching. We provide geometric insight and discuss implications for future discrimination-based evaluation metrics.

Problem

• The study examines PDS sensitivity to distance metrics in high-dimensional gene-expression data.
• It analyzes how scaling affects perturbation discrimination score behavior and reliability.
• The research compares geometric differences between L1/L2 and cosine-based PDS measures.

Innovation

• PDS sensitivity to distance metric choice
• PDS sensitivity to effect scaling
• Geometric insight for evaluation metrics