🤖 AI Summary
为提高LLM软件系统的评估可靠性，提出了一种端到端评估流程，通过创建评估清单并结合学习聚合方法，以增强评估一致性和准确性。
📝 Abstract
LLM-based software systems increasingly require effective "evals" as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.