Structured Matrix Scaling for Multi-Class Calibration

2025-11-05

AI Summary

Existing post-hoc calibration methods for multi-class classifiers—particularly logistic regression–based approaches—suffer from overfitting due to excessive parameters and limited calibration data. Method: We propose Structured Matrix Scaling (SMS), a principled calibration framework built upon multinomial logistic regression, incorporating structured matrix regularization (e.g., low-rank or diagonal-plus-low-rank constraints), cross-class parameter sharing, and robust feature preprocessing. This design simultaneously enhances expressivity and controls variance. Contribution/Results: SMS theoretically overcomes the representational limitations of temperature scaling and standard matrix scaling, achieving a superior bias–variance trade-off. Extensive experiments across diverse models and datasets demonstrate that SMS significantly outperforms existing logistic regression–based calibration methods, while exhibiting strong scalability and practicality. The implementation is open-sourced, establishing a new efficient and robust benchmark for probabilistic calibration.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Post-hoc recalibration methods are widely used to ensure that classifiers provide faithful probability estimates. We argue that parametric recalibration functions based on logistic regression can be motivated from a simple theoretical setting for both binary and multiclass classification. This insight motivates the use of more expressive calibration methods beyond standard temperature scaling. For multi-class calibration however, a key challenge lies in the increasing number of parameters introduced by more complex models, often coupled with limited calibration data, which can lead to overfitting. Through extensive experiments, we demonstrate that the resulting bias-variance tradeoff can be effectively managed by structured regularization, robust preprocessing and efficient optimization. The resulting methods lead to substantial gains over existing logistic-based calibration techniques. We provide efficient and easy-to-use open-source implementations of our methods, making them an attractive alternative to common temperature, vector, and matrix scaling implementations.

Problem

• Developing expressive multi-class calibration methods beyond temperature scaling
• Addressing overfitting from complex models with limited calibration data
• Managing bias-variance tradeoff through structured regularization and optimization

Innovation

• Structured regularization manages bias-variance tradeoff
• Robust preprocessing enhances multi-class calibration
• Efficient optimization enables complex calibration models