🤖 AI Summary
This study addresses the lack of a clearly defined normative role for human annotators in current reinforcement learning from human feedback (RLHF) approaches, which leads to ambiguous annotation protocols and risks of model failure. The work systematically introduces three normative role models—expansion, evidence, and authority—and argues that annotation processes should be decomposed along task dimensions to align with the most appropriate role model. Through conceptual analysis, literature review, and the construction of a normative framework, the research uncovers implicit normative assumptions underlying existing methods, clarifies the hazards arising from role confusion, and establishes clear normative criteria for selecting and designing annotation protocols. This contribution advances a new paradigm of dimension-specific annotation mechanism design in RLHF.
📝 Abstract
Preference-based alignment methods, most prominently Reinforcement Learning with Human Feedback (RLHF), use the judgments of human annotators to shape large language model behaviour. However, the normative role of these judgments is rarely made explicit. I distinguish three conceptual models of that role. The first is extension: annotators extend the system designers' own judgments about what outputs should be. The second is evidence: annotators provide independent evidence about some facts, whether moral, social or otherwise. The third is authority: annotators have some independent authority (as representatives of the broader population) to determine system outputs. I argue that these models have implications for how RLHF pipelines should solicit, validate and aggregate annotations. I survey landmark papers in the literature on RLHF and related methods to illustrate how they implicitly draw on these models, describe failure modes that come from unintentionally or intentionally conflating them, and offer normative criteria for choosing among them. My central recommendation is that RLHF pipeline designers should decompose annotation into separable dimensions and tailor each pipeline to the model most appropriate for that dimension, rather than seeking a single unified pipeline.