Towards Accurate and Calibrated Classification: Regularizing Cross-Entropy From A Generative Perspective

2026-04-08

AI Summary

Deep neural networks often exhibit overconfidence in classification tasks due to overfitting on the negative log-likelihood, making it challenging to simultaneously achieve high accuracy and well-calibrated confidence. This work proposes the Generalized Cross-Entropy (GCE) loss function, which integrates the strengths of generative and discriminative modeling by optimizing class-conditional likelihood while incorporating a class-level confidence regularization term. GCE is the first method to improve calibration without sacrificing accuracy. When combined with Adaptive Temperature Scaling (ATS) as a post-processing step, GCE consistently outperforms standard cross-entropy and Focal Loss across CIFAR-10/100, Tiny-ImageNet, and medical imaging benchmarks, demonstrating particularly strong performance under long-tailed data distributions.

Abstract

Accurate classification requires not only high predictive accuracy but also well-calibrated confidence estimates. Yet, modern deep neural networks (DNNs) are often overconfident, primarily due to overfitting on the negative log-likelihood (NLL). While focal loss variants alleviate this issue, they typically reduce accuracy, revealing a persistent trade-off between calibration and predictive performance. Motivated by the complementary strengths of generative and discriminative classifiers, we propose Generative Cross-Entropy (GCE), which maximizes $p(x|y)$ and is equivalent to cross-entropy augmented with a class-level confidence regularizer. Under mild conditions, GCE is strictly proper. Across CIFAR-10/100, Tiny-ImageNet, and a medical imaging benchmark, GCE improves both accuracy and calibration over cross-entropy, especially in the long-tailed scenario. Combined with adaptive piecewise temperature scaling (ATS), GCE attains calibration competitive with focal-loss variants without sacrificing accuracy.

Problem

• classification calibration
• overconfidence
• accuracy-calibration trade-off
• deep neural networks
• confidence estimation

Innovation

• Generative Cross-Entropy
• calibration
• confidence regularization
• long-tailed classification
• adaptive temperature scaling