Calibration without Ground Truth

2026-01-27

AI Summary

This work proposes an unsupervised post-processing framework to enhance both the performance and calibration of a strong but miscalibrated primary model by leveraging a weak yet well-calibrated reference model, particularly in settings where ground-truth labels are unavailable. The approach introduces and formalizes the notion of “non-mutual calibration,” establishing a novel connection to arbitrage theory from economics, and provides rigorous worst-case loss guarantees under any proper loss function. Built upon Bregman projections and proper scoring rules, the algorithm efficiently achieves unsupervised calibration. Empirical evaluations demonstrate that the method consistently reduces proper loss and calibration error across large language models of varying scales, achieving performance comparable to supervised baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Villalobos et al. [2024] predict that publicly available human text will be exhausted within the next decade. Thus, improving models without access to ground-truth labels becomes increasingly important. We propose a label-free post-processing framework that improves a strong but miscalibrated model using a weaker yet better-calibrated reference. Our framework guarantees a strict performance improvement under any proper loss. Our approach is based on a characterization of when strict improvement is possible: when the strong and reference models are not mutually calibrated. We formalize this condition, connect it to arbitrage and no-trade results from economics, and develop an efficient Bregman projection algorithm that guarantees worst-case loss reduction without labels. Experiments on representative LLMs across varying scales demonstrate that our label-free method significantly reduces proper losses and calibration errors, achieving performance competitive with supervised baselines.

Problem

• calibration
• ground truth
• label-free
• proper loss
• model improvement

Innovation

• label-free calibration
• mutual calibration
• Bregman projection
• proper loss
• no-ground-truth learning