SoftMCC: An MCC-Brier Calibration Bridge for Threshold-Free Model Selection under Class Imbalance

2026-08-09

AI Summary

This work addresses the instability of validation rankings in class-imbalanced settings caused by threshold dependence in conventional Matthews correlation coefficient (MCC)–based model selection. The authors propose SoftMCC, a threshold-free framework that constructs an MCC-based validation metric using probabilistic outputs. Under perfect calibration, SoftMCC unifies MCC with the Brier skill score and introduces a covariance-normalized measure of association between probabilities and labels, which retains Pearson’s boundedness and reduces to hard-prediction MCC in the deterministic limit. Integrating probabilistic confusion counts, temperature scaling, and tie-aware shared pool selection, SoftMCC achieves the best average rank (2.31) and highest Kendall’s W (0.659) across 18 experimental settings, significantly outperforming AUPRC and MCC@0.5 in ranking stability. However, it remains sensitive to miscalibration and yields only modest gains in test-set MCC.

Abstract

Model selection for imbalanced binary classification often uses the Matthews correlation coefficient (MCC), but thresholding makes validation rankings threshold-dependent. SoftMCC is a post-training MCC validation framework on established probability-valued confusion counts, coupling an MCC-specific calibrated identity with a tie-aware, shared-pool selection protocol. Its core score is a covariance-normalized probability-label association, reduces exactly to MCC for hard predictions, and is Pearson-bounded. Under perfect population calibration it equals the Brier skill score with identical candidate ordering; outside that regime the gap does not identify calibration error. Across 18 settings with 12 duplicate-safe grouped repeats, SoftMCC attains the best stability mean rank (2.31) and highest mean tie-corrected Kendall's W (0.659), with a significant Friedman test (p=0.007); Nemenyi analysis separates it from AUPRC and MCC@0.5, while 14-source-family sensitivity retains only the latter. Selected-model utility shows no advantage. Three of six prespecified comparisons have negative mean test-MCC differences, only F1@best survives Holm correction (p=0.014), and the dataset-level test is not significant (p=0.117). Label permutation lowers mean W to 0.092; temperature scaling shifts SoftMCC rankings (mean Spearman 0.851) whereas rank-based and threshold-optimized metrics remain invariant. SoftMCC is a calibration-sensitive MCC-family selector with bounded stability and utility evidence.

Problem

• class imbalance
• model selection
• Matthews correlation coefficient
• threshold dependence
• calibration

Innovation

• SoftMCC
• Matthews correlation coefficient
• probability calibration
• threshold-free model selection
• class imbalance