From Preferences to Principles: Rubric-Based Alignment for Grounded Knowledge Answers

2026-08-24

AI Summary

本文通过基于评分标准的奖励框架，利用检索到的证据生成多维度的质量评估，以改进开放领域问题回答的质量。

Abstract

Designing effective reward signals for open-domain question answering is challenging because high-quality responses must simultaneously satisfy multiple aspects of answer quality that are difficult to capture with a holistic scalar objective. We introduce a rubric-based reward framework that generates query-specific rubrics grounded in retrieved evidence and decomposed into multiple quality dimensions, providing fine-grained supervision during post-training. Averaged across three evaluation axes (composition, grounding, and instruction-following), our approach improves over the instruction-tuned baseline by 6.5% and over flat rubric variants by 4%, with consistent gains across all evaluation datasets. Conditioning rubrics on retrieved evidence improves factual support, while decomposing rubrics into quality-specific dimensions further improves coherence, organization, and adherence to query requirements. Our results show that grounded, multi-dimensional rubrics provide more effective reward supervision for complex open-domain question answering.

Problem

• reward signals
• open-domain question answering
• answer quality
• rubric-based reward

Innovation

• rubric-based reward
• query-specific rubrics
• multi-dimensional quality
• retrieved evidence
• fine-grained supervision