Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknown

2026-04-13

AI Summary

This work addresses the limited reliability of deep neural networks in high-stakes scenarios due to poor confidence calibration, a challenge exacerbated by the difficulty of simultaneously achieving strong classification performance and well-calibrated predictions. To this end, we propose Socrates Loss, which uniquely incorporates an explicit unknown class into the loss function and jointly optimizes classification accuracy and confidence calibration through a dynamic uncertainty-aware penalty mechanism. By integrating a unified loss formulation, principled uncertainty modeling, and theoretical regularization analysis, our approach ensures training stability and mitigates overfitting. Extensive experiments demonstrate that Socrates Loss consistently achieves a superior trade-off between accuracy and calibration across four benchmark datasets and multiple network architectures, while also enabling faster convergence.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Deep neural networks, despite their high accuracy, often exhibit poor confidence calibration, limiting their reliability in high-stakes applications. Current ad-hoc confidence calibration methods attempt to fix this during training but face a fundamental trade-off: two-phase training methods achieve strong classification performance at the cost of training instability and poorer confidence calibration, while single-loss methods are stable but underperform in classification. This paper addresses and mitigates this stability-performance trade-off. We propose Socrates Loss, a novel, unified loss function that explicitly leverages uncertainty by incorporating an auxiliary unknown class, whose predictions directly influence the loss function and a dynamic uncertainty penalty. This unified objective allows the model to be optimized for both classification and confidence calibration simultaneously, without the instability of complex, scheduled losses. We provide theoretical guarantees that our method regularizes the model to prevent miscalibration and overfitting. Across four benchmark datasets and multiple architectures, our comprehensive experiments demonstrate that Socrates Loss consistently improves training stability while achieving more favorable accuracy-calibration trade-off, often converging faster than existing methods.

Problem

• confidence calibration
• classification
• training stability
• accuracy-calibration trade-off
• deep neural networks

Innovation

• confidence calibration
• unified loss function
• unknown class
• uncertainty modeling
• training stability