Attention Limited Reward Learning

2026-07-05

AI Summary

This work addresses a fundamental limitation in conventional reward modeling: its inability to distinguish between cases where options have genuinely similar utilities and those where human annotators struggle to differentiate due to limited attention. To resolve this, the authors introduce a rational inattention framework, modeling human feedback as a perception process constrained by attentional resources. They propose an attention-limited pairwise comparison generative model and demonstrate that the standard Bradley–Terry model fails to disentangle the confounding effects of true reward, attention allocation, and default response bias. Both theoretical analysis and experiments reveal that the efficacy of preference learning hinges on the amount of attentional information per label rather than sheer label quantity. Applying their approach to Chatbot Arena data uncovers significant cyclic preferences, and perceptual experiments confirm that reaction times and eye-tracking behaviors encode value differences not captured by explicit labels.

Abstract

Pairwise human comparisons are a primary interface through which modern AI systems learn human preferences. RLHF and related alignment pipelines typically model such comparisons with Bradley--Terry log-odds, where choice probabilities are governed by latent reward differences. This paper examines what this assumption misses through a reduced-form model motivated by rational inattention, in which each label is generated by a low-capacity evaluation channel. The model separates two forms of ambiguity that standard reward modeling tends to conflate: a comparison may be difficult because the two candidates are genuinely close in value, or because the relevant distinction is hard to detect under limited attention. We show that limited attention can fundamentally distort what pairwise comparisons reveal. In particular, passive comparison data cannot generally distinguish reward, attention, and default tendencies, and heterogeneous attention can make standard Bradley--Terry reward modeling recover misleading rankings. Our analysis shows that learning is governed not by the raw number of labels, but by the amount of attended information each label carries. A case study on human votes over language-model pairs from Chatbot Arena exhibits the predicted signature, a cyclic component of the comparison data that exceeds sampling noise and that no scalar reward can represent; a second case study on perceptual comparisons shows that response times and gaze carry gap information that the labels do not. This perspective suggests that human feedback should be treated not as direct revealed preference, but as an attention-limited measurement process: a weak preference signal may reflect hidden evaluation difficulty rather than genuine indifference.

Problem

• reward learning
• pairwise comparisons
• rational inattention
• preference ambiguity
• attention limitation

Innovation

• rational inattention
• preference learning
• Bradley-Terry model
• attention-limited feedback
• revealed preference