A Survey on Rubric-Guided Reinforcement Learning for Language Models

2026-08-27

AI Summary

本文探讨了通过引入结构化、可解释的评估标准（即评分细则）来改进基于人类反馈的强化学习方法，以解决传统方法中奖励信号缺乏解释性和多维度的问题。

Abstract

Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. However, traditional RLHF relies on scalar reward signals that lack interpretability and fail to capture the multifaceted nature of response quality. Rubric-guided reinforcement learning addresses these limitations by introducing structured, interpretable evaluation criteria, or rubrics, as the backbone of reward design, feedback generation, and policy optimization. In this survey, we introduce a Bayesian framework that defines constitutions as prior distributions $P(R)$ over evaluation criteria and rubrics as conditional instantiations $R_x \sim P(R|x)$. Under this unified view, we present a taxonomy of rubric-guided RL along the prior-posterior axis, covering constitutional AI, instance-specific rubrics, process-level supervision, self-evolving rubrics, and their agentic and multimodal extensions. Furthermore, as rubrics are natural-language artifacts, we present a linguistic analysis of how granularity trade-offs, semantic drift, and linguistic reward hacking impact alignment reliability, identifying key open problems for future research.

Problem

• Reinforcement Learning
• Human Feedback
• Language Models
• Evaluation Criteria
• Interpretability

Innovation

• Bayesian framework
• rubric-guided reinforcement learning
• prior distributions over evaluation criteria
• self-evolving rubrics
• alignment reliability