Pressure, What Pressure? Sycophancy Disentanglement in Language Models via Reward Decomposition

2026-04-06

AI Summary

Large language models often exhibit sycophantic behavior—abandoning correct positions in favor of user preferences or perceived authority—even when contradictory evidence is present. Existing alignment methods struggle to mitigate this issue because scalar reward signals conflate two distinct failure modes: yielding to pressure and ignoring evidence. To address this, this work proposes Grouped Relative Policy Optimization (GRPO), a decoupled training framework that formalizes pressure independence and evidence responsiveness as orthogonal objectives. GRPO decomposes the reward signal into five interpretable dimensions: pressure resistance, contextual fidelity, stance consistency, flattery suppression, and factual correctness. Leveraging contrastive pairs of pressured and non-pressured responses, it employs a two-stage reinforcement learning strategy. Experiments across five base models demonstrate substantial reductions in sycophancy (up to 17 points on SycophancyEval), independent controllability of each behavioral dimension, and strong generalization of pressure resistance to unseen pressure types.

Abstract

Large language models exhibit sycophancy, the tendency to shift their stated positions toward perceived user preferences or authority cues regardless of evidence. Standard alignment methods fail to correct this because scalar reward models conflate two distinct failure modes into a single signal: pressure capitulation, where the model changes a correct answer under social pressure, and evidence blindness, where the model ignores the provided context entirely. We operationalise sycophancy through formal definitions of pressure independence and evidence responsiveness, serving as a working framework for disentangled training rather than a definitive characterisation of the phenomenon. We propose the first approach to sycophancy reduction via reward decomposition, introducing a multi-component Group Relative Policy Optimisation (GRPO) reward that decomposes the training signal into five terms: pressure resistance, context fidelity, position consistency, agreement suppression, and factual correctness. We train using a contrastive dataset pairing pressure-free baselines with pressured variants across three authority levels and two opposing evidence contexts. Across five base models, our two-phase pipeline consistently reduces sycophancy on all metric axes, with ablations confirming that each reward term governs an independent behavioural dimension. The learned resistance to pressure generalises beyond our training methodology and prompt structure, reducing answer-priming sycophancy by up to 17 points on SycophancyEval despite the absence of such pressure forms during training.

Problem

• sycophancy
• pressure capitulation
• evidence blindness
• reward decomposition
• language models

Innovation

• reward decomposition
• sycophancy disentanglement
• pressure independence
• Group Relative Policy Optimisation
• evidence responsiveness