Learning When to Trust in Contextual Bandits

2026-03-08

AI Summary

This work addresses a critical limitation in traditional robust reinforcement learning, which typically assumes evaluators are either globally trustworthy or fully adversarial, thereby failing to handle “contextual sycophancy”—a scenario where evaluators strategically bias their feedback in specific contexts. To tackle this, the paper proposes the CESA-LinUCB algorithm, which formalizes contextual sycophancy as a failure mode and introduces the notion of “contextual objective decoupling.” By online learning high-dimensional, context-dependent trust boundaries for each evaluator, CESA-LinUCB dynamically identifies their locally reliable regions without requiring global reliability assumptions. Integrated within the LinUCB framework, the method combines context-aware trust modeling with adaptive evaluator selection. Theoretically, it is shown to recover the true reward signal even under contextual adversarial conditions and achieves a sublinear regret bound of Õ(√T).

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Standard approaches to Robust Reinforcement Learning assume that feedback sources are either globally trustworthy or globally adversarial. In this paper, we challenge this assumption and we identify a more subtle failure mode. We term this mode as Contextual Sycophancy, where evaluators are truthful in benign contexts but strategically biased in critical ones. We prove that standard robust methods fail in this setting, suffering from Contextual Objective Decoupling. To address this, we propose CESA-LinUCB, which learns a high-dimensional Trust Boundary for each evaluator. We prove that CESA-LinUCB achieves sublinear regret $\tilde{O}(\sqrt{T})$ against contextual adversaries, recovering the ground truth even when no evaluator is globally reliable.

Problem

• Contextual Bandits
• Robust Reinforcement Learning
• Contextual Sycophancy
• Trustworthy Feedback
• Adversarial Evaluators

Innovation

• Contextual Sycophancy
• Trust Boundary
• Robust Reinforcement Learning
• Contextual Bandits
• Sublinear Regret