Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL

2026-06-12

AI Summary

This work addresses the significant reflection bias exhibited by large language model agents, which struggle to accurately evaluate their own performance even with environmental feedback. Standard reinforcement learning approaches often fail to mitigate this issue due to misaligned credit assignment. To overcome these limitations, the authors propose RefGRPO, a novel method that leverages the discrepancy between an agent’s self-reflection and actual outcomes to construct an unsupervised calibration signal. This approach introduces a calibration reward term that requires neither additional annotations nor a separate reward model, and integrates a dynamically scheduled coefficient with a pseudo-reward-based self-improvement mechanism. Evaluated across five text-to-SQL benchmarks, RefGRPO reduces the underconfident self-assessment rate from 44.4% to 7.7% and improves task accuracy from 75.1% to 76.5%, substantially enhancing both self-evaluation capability and decision reliability.

Abstract

LLMs are increasingly deployed as agents that interact with external environments and observe feedback such as execution results, error messages, and tool outputs. A well-functioning agent should be able to leverage this feedback to accurately assess its own performance. Yet we find a persistent reflection gap: LLM agents tend to mis-assess their own outputs after observing concrete environment feedback -- even for questions they correctly answered -- and standard RL barely helps due to a credit-assignment mismatch. To close this gap, we propose RefGRPO, a simple yet effective fix that augments standard RL algorithms with two key ingredients: a free calibration bonus computed by contrasting the agent's own reflection with the actual outcome (requiring no additional reward model, LLM judge, or external annotation), and a dynamic schedule on its coefficient. Compared to standard RL baselines, our method simultaneously improves reflection calibration (e.g., reduces underconfidence rate $44.4\% \to 7.7\%$) and task accuracy (e.g., $75.1\% \to 76.5\%$) on text-to-SQL across five benchmarks. The resulting calibrated reflection turns the agent into its own verifier grounded in environment feedback, which further enables (i) better self-improvement that uses reflections as pseudo-rewards without outcome supervision, and (ii) more effective test-time selective prediction by committing only to rollouts flagged as correct.

Problem

• reflection gap
• agentic RL
• self-assessment
• calibration
• LLM agents

Innovation

• reflection gap
• calibration bonus
• agentic RL
• self-verification
• dynamic scheduling