Mitigating Cognitive Bias in RLHF by Altering Rationality

2026-05-07

AI Summary

This work addresses the vulnerability of reinforcement learning from human feedback (RLHF) to cognitive biases, which can lead reward models to learn irrational preferences. The authors propose a novel approach that models the rationality parameter β as a dynamic variable dependent on both context and annotator identity. By integrating an LLM-as-judge mechanism to assess the likelihood of cognitive bias in pairwise comparison data, the method adaptively reweights preference samples during training. This is the first framework to dynamically model rationality in reward learning, significantly enhancing robustness against biased feedback. Empirical results demonstrate that even when fine-tuned on strongly biased preference data, the approach yields downstream policy models that better align with rational human behavior.

Abstract

How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise comparisons, this learning depends on an assumed relationship between latent reward differences and observed preferences, typically modeled using a Boltzmann formulation in which a rationality parameter beta informs how consistently preferences reflect reward differences. In practice, beta is typically treated as a fixed constant that reflects assumed uniform annotator reliability. However, human feedback is not this simplistic in practice: real human judgments are shaped by cognitive biases, leading to systematic deviations from reward-consistent behavior that arise contextually. To address this, we treat rationality as context- and annotation-dependent. We design an approach to dynamically adjust the rationality parameter beta during reward learning using an LLM-as-judge to assess the likely presence of cognitive biases. This approach effectively downweights comparisons that are likely to reflect biased or unreliable judgments. Empirically, we show that this approach learns a more rational downstream model, even when finetuning on datasets with strongly biased preferences.

Problem

• cognitive bias
• reinforcement learning from human feedback
• reward modeling
• rationality
• human preferences

Innovation

• cognitive bias
• RLHF
• rationality parameter
• LLM-as-judge
• dynamic beta adjustment