From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimation

2026-07-14

AI Summary

This work addresses the unreliability of confidence estimates in large language models for quantitative prediction, stemming from hallucination and overconfidence. To tackle this issue, the authors propose the CARE-PPO framework, which uniquely reconfigures the Critic component in Proximal Policy Optimization (PPO) as a confidence estimator and introduces an error-aware confidence alignment reward function to jointly optimize prediction accuracy and confidence reliability. Built upon an Actor-Critic architecture and leveraging fine-tuned Qwen-3 models (4B/8B) with out-of-distribution generalization strategies, the method significantly outperforms baseline approaches—such as logit-based and verbalized confidence methods—on medical and financial tasks, demonstrating consistent improvements in predictive performance, calibration quality, and cross-domain generalization.

Abstract

LLMs can perform language-based quantitative prediction from unstructured inputs, but remain susceptible to hallucinations and overconfident errors, making it critical to know not only what a model predicts, but when its predictions can be trusted. We introduce CARE-PPO, a reinforcement learning framework that establishes a connection between loss prediction for uncertainty estimation and actor-critic PPO fine-tuning, enabling joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction. CARE-PPO uses a Confidence-Aligned Reward for Estimation, defined as a function of prediction error, to provide dense error-aware feedback to the actor while inducing the critic to learn a value function aligned with prediction quality. During inference, we repurpose the critic as a confidence estimator. Across two real-world tasks in healthcare and finance and two Qwen-3 model scales (4B and 8B), CARE-PPO achieves strong quantitative prediction performance, while producing significantly better-aligned confidence estimates through the critic than logit-based and verbalized baselines. These gains persist under realistic out-of-distribution settings across domains, spanning linguistic and domain shifts. Finally, CARE-PPO reduces task-specific overfitting on general instruction-following prompts, consistent with the broader generalization advantages of RL fine-tuning over supervised approaches.

Problem

• hallucination
• overconfidence
• confidence estimation
• quantitative prediction
• trustworthiness

Innovation

• CARE-PPO
• confidence estimation
• actor-critic PPO
• language-based quantitative prediction
• uncertainty-aware reinforcement learning