Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?

2026-06-30

AI Summary

This work addresses the susceptibility of large language model (LLM) agents to systematic biases in evaluator feedback during policy optimization, which can induce preference coupling and distort the policy distribution. To mitigate this issue, the study introduces probabilistic calibration into the LLM evaluation feedback loop for the first time, proposing a lightweight Calibrated TTRL protocol. This approach calibrates pairwise judgments from evaluators and integrates them with a confidence-weighted TTRL update mechanism to effectively alleviate preference coupling. Experiments employing DeepSeek-V4-Pro as the actor and GLM5.2 as the evaluator—under symmetric learning rate control—demonstrate a 20–49% reduction in the coupling coefficient γ and a 45–67% decrease in Jensen–Shannon divergence, substantiating the method’s efficacy in preserving policy fidelity.

Abstract

When large language model (LLM) agents adapt their behavior through evaluator feedback, systematic evaluator biases propagate into the agent's learned strategy distribution - a phenomenon termed evaluator preference coupling. Prior work has documented this coupling and established a diagnostic framework (EPC) to measure it, but has not investigated whether calibration techniques can mitigate the effect. We present the first study of evaluator calibration as mitigation: applying probability calibration to the evaluator's pairwise judgments to reduce spurious preference propagation. In a controlled within-subjects experiment (N=5) comparing standard binary TTRL (win/loss) with confidence-calibrated TTRL (probability-weighted updates) using DeepSeek-V4-Pro as executor and GLM5.2 as evaluator, we find that calibration reduces the coupling coefficient gamma by 20-49% and Jensen-Shannon divergence by 45-67%. A symmetric-LR control confirms the effect is not due to reduced update asymmetry. We release the calibrated TTRL protocol and recommend it as a lightweight mitigation for LLM-as-judge deployment pipelines.

Problem

• preference coupling
• probability calibration
• LLM agent
• evaluator bias
• feedback loop

Innovation

• probability calibration
• preference coupling
• TTRL
• LLM-as-judge
• feedback loop