Principled Algorithms for Optimizing Generalized Metrics in Binary Classification

2025-12-28

AI Summary

Optimizing generalized evaluation metrics—such as Fβ, accuracy parity (AM), and Jaccard—under class imbalance and asymmetric misclassification costs remains challenging, as these metrics are non-decomposable and non-differentiable, precluding direct empirical risk minimization without probabilistic calibration or threshold tuning. Method: This paper introduces the first unified cost-sensitive learning framework that bypasses both probability estimation and threshold optimization. At its core lies a novel theory for constructing H-consistent surrogate losses with finite-sample generalization bounds. We further propose METRO, an algorithm that achieves provably optimal empirical risk minimization over arbitrary hypothesis classes. Contribution/Results: Evaluated on multiple multiclass imbalanced benchmarks, our method consistently outperforms state-of-the-art approaches. Empirical results validate its superior convergence behavior and generalization performance, demonstrating that theoretical guarantees translate effectively into practical gains.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In applications with significant class imbalance or asymmetric costs, metrics such as the $F_β$-measure, AM measure, Jaccard similarity coefficient, and weighted accuracy offer more suitable evaluation criteria than standard binary classification loss. However, optimizing these metrics present significant computational and statistical challenges. Existing approaches often rely on the characterization of the Bayes-optimal classifier, and use threshold-based methods that first estimate class probabilities and then seek an optimal threshold. This leads to algorithms that are not tailored to restricted hypothesis sets and lack finite-sample performance guarantees. In this work, we introduce principled algorithms for optimizing generalized metrics, supported by $H$-consistency and finite-sample generalization bounds. Our approach reformulates metric optimization as a generalized cost-sensitive learning problem, enabling the design of novel surrogate loss functions with provable $H$-consistency guarantees. Leveraging this framework, we develop new algorithms, METRO (Metric Optimization), with strong theoretical performance guarantees. We report the results of experiments demonstrating the effectiveness of our methods compared to prior baselines.

Problem

• Optimizing generalized metrics in binary classification
• Addressing computational and statistical challenges in metric optimization
• Developing algorithms with theoretical guarantees for generalized metrics

Innovation

• Reformulates metric optimization as cost-sensitive learning problem
• Introduces novel surrogate loss functions with H-consistency guarantees
• Develops METRO algorithm with strong theoretical performance guarantees