Weighted MCC: A Robust Measure of Multiclass Classifier Performance for Observations with Individual Weights

2025-12-23

AI Summary

Conventional binary and multiclass evaluation metrics ignore sample-weight heterogeneity, failing to reflect model performance on high-weight samples. Method: We propose the Weighted Matthews Correlation Coefficient (WMCC) and its multiclass extension (WMC³), establishing the first rigorous, robust theoretical framework for weighted MCC. Grounded in weighted confusion matrix modeling, WMCC generalizes MCC as the natural weighted form of the Pearson correlation coefficient. Contribution/Results: We prove that WMCC exhibits ε-level stability under weight perturbations in binary classification, while WMC³ achieves ε²-level stability in multiclass settings. Empirical results demonstrate that WMCC/WMC³ effectively discriminate among weight-sensitive models—whereas standard unweighted metrics fail entirely. This work introduces a theoretically grounded and practically effective evaluation standard for weighted classification tasks.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Decision/Utility Theory
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Several performance measures are used to evaluate binary and multiclass classification tasks. But individual observations may often have distinct weights, and none of these measures are sensitive to such varying weights. We propose a new weighted Pearson-Matthews Correlation Coefficient (MCC) for binary classification as well as weighted versions of related multiclass measures. The weighted MCC varies between $-1$ and $1$. But crucially, the weighted MCC values are higher for classifiers that perform better on highly weighted observations, and hence is able to distinguish them from classifiers that have a similar overall performance and ones that perform better on the lowly weighted observations. Furthermore, we prove that the weighted measures are robust with respect to the choice of weights in a precise manner: if the weights are changed by at most $ε$, the value of the weighted measure changes at most by a factor of $ε$ in the binary case and by a factor of $ε^2$ in the multiclass case. Our computations demonstrate that the weighted measures clearly identify classifiers that perform better on higher weighted observations, while the unweighted measures remain completely indifferent to the choices of weights.

Problem

• Develops weighted MCC for multiclass classification with individual observation weights
• Ensures robust performance measure sensitivity to varying observation weights
• Distinguishes classifiers based on performance on high versus low weighted observations

Innovation

• Weighted MCC adapts Pearson-Matthews Correlation Coefficient for individual observation weights
• Weighted measures robust to weight changes with bounded error factors
• Weighted MCC distinguishes classifiers by performance on highly weighted observations