Rethinking Post-Hoc Calibration in Semantic Segmentation

2026-07-02

AI Summary

Semantic segmentation models often suffer from poor confidence calibration, and existing post-hoc calibration methods frequently lead to unstable calibration or performance degradation due to their neglect of representational dependencies in dense prediction and misalignment with segmentation objectives. This work proposes a translation-invariant, decision-preserving post-processing calibration framework that enhances confidence reliability without retraining while maintaining segmentation quality. The approach employs class-conditional affine calibrators, incorporates translation invariance to eliminate logit shift effects, and enforces constraints that preserve either the argmax or the ranking of predictions, effectively balancing calibration accuracy and segmentation performance. Experiments demonstrate that the method significantly improves calibration metrics across both natural and medical image benchmarks, as well as under covariate shift, without compromising segmentation fidelity.

Abstract

Reliable confidence estimates are essential in semantic segmentation, especially in safety-critical settings where overconfident errors can mislead downstream decisions. Yet modern segmentation models often remain miscalibrated. Post-hoc calibration offers a practical way to correct confidence estimates without retraining the segmentation model, but its use in dense prediction raises structural issues that are often overlooked. We study two such issues. First, adding a constant to all logits leaves the softmax probabilities unchanged, but several standard calibrators can still depend on this arbitrary offset. As a result, two logit representations encoding the same predictive distribution may yield different calibrated probabilities. We define translation-invariant (TI) calibrators as those whose outputs are unchanged under such shifts, characterize which common calibrators satisfy this property, and construct TI counterparts of shift-sensitive calibrators to isolate the effect of removing representation dependence. Second, post-hoc calibration is typically fitted by minimizing a likelihood-based objective, whereas segmentation models are trained with task-specific metrics such as Dice. This mismatch can cause calibration to alter class orderings and degrade the deployed segmentation map. We study decision-preserving calibration under argmax- and order-preservation constraints. Since enforcing these constraints collapses affine softmax calibrators to temperature scaling, we introduce class-conditional affine calibrators that can be made argmax- or order-preserving while retaining greater expressivity, allowing us to quantify the calibration-segmentation trade-off induced by decision preservation. Across natural-image and medical segmentation benchmarks, and under corruption-based covariate shift, matched comparisons show that TI variants generally improve calibration metrics, while decision-preserving variants prevent segmentation degradation and retain strong calibration performance. These results provide practical design principles for well-defined post-hoc calibration pipelines in semantic segmentation.

Problem

• post-hoc calibration
• semantic segmentation
• translation invariance
• decision preservation
• confidence calibration

Innovation

• translation-invariant calibration
• decision-preserving calibration
• semantic segmentation
• post-hoc calibration
• class-conditional affine calibrators