Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques

2025-12-09

AI Summary

To address the poor probability calibration of multiclass classifiers and the limited scalability and performance of existing isotonic regression methods, this paper proposes two normalization-aware isotonic calibration approaches: (1) NA-FIR, which explicitly incorporates the probability simplex constraint into isotonic optimization; and (2) SCIR, which employs cumulative bivariate modeling. Both methods overcome the suboptimality inherent in conventional one-vs-rest (OvR) isotonic regression while preserving theoretical soundness, enhanced interpretability, and practical applicability. Extensive experiments on multiclass text and image classification benchmarks demonstrate that the proposed methods significantly reduce negative log-likelihood (NLL) and expected calibration error (ECE), consistently outperforming state-of-the-art parametric and nonparametric calibration techniques across all evaluation metrics.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text

Abstract

Accurate and reliable probability predictions are essential for multi-class supervised learning tasks, where well-calibrated models enable rational decision-making. While isotonic regression has proven effective for binary calibration, its extension to multi-class problems via one-vs-rest calibration produced suboptimal results when compared to parametric methods, limiting its practical adoption. In this work, we propose novel isotonic normalization-aware techniques for multiclass calibration, grounded in natural and intuitive assumptions expected by practitioners. Unlike prior approaches, our methods inherently account for probability normalization by either incorporating normalization directly into the optimization process (NA-FIR) or modeling the problem as a cumulative bivariate isotonic regression (SCIR). Empirical evaluation on a variety of text and image classification datasets across different model architectures reveals that our approach consistently improves negative log-likelihood (NLL) and expected calibration error (ECE) metrics.

Problem

• Extends isotonic regression to multi-class calibration
• Addresses probability normalization in calibration methods
• Improves calibration metrics across diverse datasets

Innovation

• Novel isotonic normalization-aware techniques for multiclass calibration
• Incorporates normalization into optimization process (NA-FIR)
• Models problem as cumulative bivariate isotonic regression (SCIR)