Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization

2026-05-20

AI Summary

This work addresses the instability in post-training performance caused by existing semantic entropy–based uncertainty estimation, which struggles to reliably distinguish informative signals from noise during policy optimization. The study reveals, for the first time, two critical limitations of semantic entropy: its anisotropy and insufficient calibration. To overcome these issues, the authors propose the Geometry-aware Calibrated Policy Optimization (GCPO) framework, which leverages a geometry-aware measure of semantic disagreement to accurately capture gradient variability and dynamically calibrates uncertainty using reward signals to filter high-quality learning signals. Experimental results across multiple benchmarks demonstrate that GCPO significantly enhances post-training performance, confirming the effectiveness and robustness of the proposed approach in modeling optimization dynamics and uncertainty.

Abstract

Post-training has become central to improving reasoning and alignment in large language models, where critic-free models enable scalable learning from model-generated outputs but lack principled mechanisms to distinguish informative from noisy signals. Recent approaches leverage response-level measures as uncertainty signals to regulate group-based optimization methods such as GRPO. Yet their empirical success remains unstable and unclear in how they influence optimization dynamics. In this paper, we provide, to our knowledge, the first principled formulation that interprets uncertainty signals as mechanisms for characterizing and regulating gradient variance and learning signal quality. Based on both empirical and theoretical analysis, we identify two critical gaps of current entropy-based estimators: The anisotropic gap and The calibration gap. Motivated by this analysis, we propose Geometric-aware Calibrated Policy Optimization (GCPO), a novel framework integrating geometry-aware measures to capture semantic disagreement with reward-based calibration to align uncertainty with learning signal strength. Experiments on multiple benchmarks show that our approach more faithfully tracks gradient variability and consistently improves post-training performance. Our results highlight the importance of designing uncertainty signals that are aligned with optimization dynamics, offering a principled perspective for robust post-training.

Problem

• semantic entropy
• uncertainty estimation
• policy optimization
• post-training
• gradient variance

Innovation

• semantic entropy
• gradient variance
• geometry-aware uncertainty
• calibrated policy optimization
• post-training