Revisiting Reweighted Risk for Calibration: AURC, Focal Loss, and Inverse Focal Loss

2025-05-29

AI Summary

This work investigates the theoretical relationship between weighted risk functions—including focal loss, inverse focal loss, and the Area-Under-the-Reliability-Curve (AURC)—and model calibration error. To bridge selective classification and calibration objectives, we propose a differentiable AURC optimization framework. We introduce SoftRank to enable end-to-end AURC minimization and support flexible confidence score function (CSF) modeling. Theoretical analysis reveals that inverse focal loss aligns more closely with calibration objectives than focal loss. Extensive experiments across multiple datasets and model architectures demonstrate that our method significantly improves class-wise calibration performance, achieving state-of-the-art (SOTA) results—outperforming standard cross-entropy and focal loss baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Several variants of reweighted risk functionals, such as focal losss, inverse focal loss, and the Area Under the Risk-Coverage Curve (AURC), have been proposed in the literature and claims have been made in relation to their calibration properties. However, focal loss and inverse focal loss propose vastly different weighting schemes. In this paper, we revisit a broad class of weighted risk functions commonly used in deep learning and establish a principled connection between these reweighting schemes and calibration errors. We show that minimizing calibration error is closely linked to the selective classification paradigm and demonstrate that optimizing a regularized variant of the AURC naturally leads to improved calibration. This regularized AURC shares a similar reweighting strategy with inverse focal loss, lending support to the idea that focal loss is less principled when calibration is a desired outcome. Direct AURC optimization offers greater flexibility through the choice of confidence score functions (CSFs). To enable gradient-based optimization, we introduce a differentiable formulation of the regularized AURC using the SoftRank technique. Empirical evaluations demonstrate that our AURC-based loss achieves competitive class-wise calibration performance across a range of datasets and model architectures.

Problem

• Analyzing reweighted risk functions for model calibration
• Linking calibration errors to weighted risk schemes
• Optimizing regularized AURC for improved calibration performance

Innovation

• Regularized AURC improves calibration error
• Differentiable AURC via SoftRank technique
• AURC shares strategy with inverse focal loss