CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

2026-05-19

AI Summary

This work addresses the limitations of existing code evaluation methods for large language models, which rely on pointwise scoring and struggle to capture subtle preference distinctions between paired responses. To overcome this, the authors propose CriterAlign, a novel framework that introduces a criterion-centric pairwise judgment mechanism. CriterAlign integrates criterion-level pairwise comparisons, tie-driven criterion refinement, swap-consistency filtering, and offline pairwise synthesis to effectively adapt scoring criteria to preference judgment tasks. Furthermore, it incorporates a Human Preference Alignment Guidance (HPAG) signal to enhance both accuracy and interpretability. Evaluated on the BigCodeReward dataset, the method significantly improves the accuracy of the Qwen2.5-VL-32B standalone evaluator from 60.4% to 66.3%, demonstrating its effectiveness in aligning model judgments with human preferences.

Abstract

Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.

Problem

• code preference judging
• pairwise evaluation
• rubric-based judging
• LLM judges
• preference prediction

Innovation

• CriterAlign
• pairwise preference prediction
• criterion-centric alignment
• Human-Preference-Aligned Guidance
• rubric-based judging