Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence

2026-07-16

AI Summary

This study addresses the challenge of automatically generating scoring rules that effectively distinguish response quality using only a single query, without relying on external annotations, preference data, or model training. To this end, the authors propose a self-evolving framework in which large language models synthesize rule-constrained response pairs, and pairwise comparisons are used to evaluate the discriminative power of candidate rules. Through iterative refinement, the framework automatically filters out non-discriminative, overly specific, or style-biased rules, retaining only high-quality scoring criteria. This approach represents the first fully unsupervised method for automatic scoring rule generation, achieving state-of-the-art average accuracy across five preference benchmarks and ranking first on six out of seven evaluation sets.

Abstract

Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.

Problem

• rubrics
• query-specific evaluation
• large language models
• synthetic supervision
• answer quality discrimination

Innovation

• rubric generation
• synthetic pairwise evidence
• query-only framework
• LLM evaluation
• self-evolving rubrics