Objective Decoupling in Social Reinforcement Learning: Recovering Ground Truth from Sycophantic Majorities

2026-02-08

AI Summary

In social reinforcement learning, human feedback is often distorted by flattery, laziness, or adversarial intent, undermining conventional alignment approaches. This work proposes Epistemic Source Alignment (ESA), a novel method that formalizes the problem of objective decoupling and introduces a “judge-the-judges” mechanism to assess the credibility of feedback sources through sparse safety axioms, rather than relying on majority consensus. Theoretically, ESA is proven to converge to the true objective; empirically, it recovers the optimal policy even when biased evaluators constitute the majority, significantly outperforming existing methods.

Technology Category

• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Contemporary AI alignment strategies rely on a fragile premise: that human feedback, while noisy, remains a fundamentally truthful signal. In this paper, we identify this assumption as Dogma 4 of Reinforcement Learning (RL). We demonstrate that while this dogma holds in static environments, it fails in social settings where evaluators may be sycophantic, lazy, or adversarial. We prove that under Dogma 4, standard RL agents suffer from what we call Objective Decoupling, a structural failure mode where the agent's learned objective permanently separates from the latent ground truth, guaranteeing convergence to misalignment. To resolve this, we propose Epistemic Source Alignment (ESA). Unlike standard robust methods that rely on statistical consensus (trusting the majority), ESA utilizes sparse safety axioms to judge the source of the feedback rather than the signal itself. We prove that this"judging the judges"mechanism guarantees convergence to the true objective, even when a majority of evaluators are biased. Empirically, we show that while traditional consensus methods fail under majority collusion, our approach successfully recovers the optimal policy.

Problem

• Objective Decoupling
• Social Reinforcement Learning
• AI Alignment
• Sycophantic Feedback
• Ground Truth Recovery

Innovation

• Objective Decoupling
• Epistemic Source Alignment
• Social Reinforcement Learning
• AI Alignment
• Sycophantic Feedback